mapreduce - Piping data into jobs in Hadoop MR/Pig -

-

i have 3 different type of jobs running on data in hdfs. these 3 jobs have run separately in current scenario. now, we want run 3 jobs piping output data of 1 job other job without writing data in hdfs improve architecture , overall performance.

any suggestions welcome scenario.

ps : oozie not fitting workflow.cascading framework ruled out because of scalability issues. thanks

hadoop inherently writes storage (e.g. hdfs) after m/r steps. if want in memory, maybe need spark.


Comments

Post a Comment

Popular posts from this blog

java - Unable to make sub reports with Jasper -

-
i using jaspersoft studio create reports. having sub report in jasper main report. problem unable make work, because if add sub report detail band of main report, sub report generated number of times, row row, entire sub report repeated in number of pages. can't put in summery band due same reason. i unable put in column footer band or other footers because displays below error net.sf.jasperreports.engine.jrexception: net.sf.jasperreports.engine.jrruntimeexception: subreport overflowed on band not support overflow. @ com.jaspersoft.studio.editor.preview.view.control.reportcontroler.fillreport(reportcontroler.java:467) @ com.jaspersoft.studio.editor.preview.view.control.reportcontroler.access$18(reportcontroler.java:442) @ com.jaspersoft.studio.editor.preview.view.control.reportcontroler$4.run(reportcontroler.java:334) @ org.eclipse.core.internal.jobs.worker.run(worker.java:54) caused by: net.sf.jasperreports.engine.jrruntimeexception: subreport overflowe...
Read more

scala - play framework: Modules were resolved with conflicting cross-version suffixes -

-
i want use elastic search play framework followed this guide here build.sbt file name := """es-with-play""" version := "1.0-snapshot" lazy val root = (project in file(".")).enableplugins(playscala) scalaversion := "2.11.1" librarydependencies ++= seq( jdbc, anorm, cache, ws, "com.clever-age" % "play2-elasticsearch" % "1.4-snapshot" ) resolvers += "sonatype oss snapshots" @ "https://oss.sonatype.org/content/repositories/snapshots" and in play console when compile code gives following errors [error] modules resolved conflicting cross-version suffixes in {file:/media/sara/new%20volume/programs/programs/play/es-with-play/}root: [error] com.jsuereth:scala-arm _2.11, _2.10 [error] com.typesafe.play:play-functional _2.11, _2.10 [error] com.typesafe.akka:akka-actor _2.11, _2.10 [error] com.typesafe.play:play-json _2.11, _2.10 [error] com.type...
Read more

Save and close a word document by giving a name in R -

-
i trying use 2 libraries ( descstool , r2wd ) in order descriptive analysis of dataset saved in folder exploration later. these steps trying library(desctools) library(r2wd) data(iris) <- ls() wrd <- getnewwrd(header=true) wdget() desc(get(i), plotit=true, wrd=wrd) wdsave(name="temp.doc", wdapp=.r2wd) wdquit() although word capturing output supposed be, cannot save word file giving name (e.g.: temp.doc). can shed light here? try library(desctools) wrd <- getnewwrd() desc(iris, wrd=wrd) wrd[["activedocument"]]$saveas2(filename="describe iris.docx") wrd$quit()
Read more