mapreduce - Piping data into jobs in Hadoop MR/Pig -

-

i have 3 different type of jobs running on data in hdfs. these 3 jobs have run separately in current scenario. now, we want run 3 jobs piping output data of 1 job other job without writing data in hdfs improve architecture , overall performance.

any suggestions welcome scenario.

ps : oozie not fitting workflow.cascading framework ruled out because of scalability issues. thanks

hadoop inherently writes storage (e.g. hdfs) after m/r steps. if want in memory, maybe need spark.


Comments

Post a Comment

Popular posts from this blog

java - Unable to make sub reports with Jasper -

-
i using jaspersoft studio create reports. having sub report in jasper main report. problem unable make work, because if add sub report detail band of main report, sub report generated number of times, row row, entire sub report repeated in number of pages. can't put in summery band due same reason. i unable put in column footer band or other footers because displays below error net.sf.jasperreports.engine.jrexception: net.sf.jasperreports.engine.jrruntimeexception: subreport overflowed on band not support overflow. @ com.jaspersoft.studio.editor.preview.view.control.reportcontroler.fillreport(reportcontroler.java:467) @ com.jaspersoft.studio.editor.preview.view.control.reportcontroler.access$18(reportcontroler.java:442) @ com.jaspersoft.studio.editor.preview.view.control.reportcontroler$4.run(reportcontroler.java:334) @ org.eclipse.core.internal.jobs.worker.run(worker.java:54) caused by: net.sf.jasperreports.engine.jrruntimeexception: subreport overflowe...
Read more

sql - The object name contains more than the maximum number of prefixes. The maximum is 2 -

-
i facing similar issue many others error message "the object name contains more maximum number of prefixes. maximum 2." my scenario is, create procedure dbo.[usp_procedure] begin declare @loadtime datetme2(7) set @loadtime = (select loadtime [linkedservername].[databasename].[schemaname].[tablename] tablename = 'xxxxxx') insert [currentserver].[schemaname].[tablename] select column1, column2, .... tablename join tablename1 on .... ... ... end the table containing loadtime present in linked server/database , important value utilized further in clause of procedure via @loadtime . query select loadtime .... works fine when try on present server individually. when run part of procedure, procedure fails above mentioned error. linked server connection set , working fine. , procedure needs run on present server, removes scope of creating procedure on linked server. hope scenario articulate. replies welcome. in advance. the line : insert [currentser...
Read more

Java Web Service Soap fault exception client -

-
i trying web service client in java using eclipse , error: exception in thread "main" javax.xml.ws.soap.soapfaultexception: message part not recognized. (does exist in service wsdl?) @ org.apache.cxf.jaxws.jaxwsclientproxy.invoke(jaxwsclientproxy.java:158) @ com.sun.proxy.$proxy26.req(unknown source) @ com.caller.main(caller.java:31) caused by: org.apache.cxf.binding.soap.soapfault: message part not recognized. (does exist in service wsdl?) @ org.apache.cxf.binding.soap.interceptor.soap11faultininterceptor.unmarshalfault(soap11faultininterceptor.java:84) @ org.apache.cxf.binding.soap.interceptor.soap11faultininterceptor.handlemessage(soap11faultininterceptor.java:51) @ org.apache.cxf.binding.soap.interceptor.soap11faultininterceptor.handlemessage(soap11faultininterceptor.java:40) @ org.apache.cxf.phase.phaseinterceptorchain.dointercept(phaseinterceptorchain.java:272) @ org.apache.cxf.interceptor.abstractfaultchaininitiatorobserver.onm...
Read more