Great to see this: > This release adds significant internal changes to Spark focused on improving performance for large scale workloads.
Great to see this: > This release adds significant internal changes to Spark focused on improving performance for large scale workloads. We looked at Spark Streaming briefly when choosing which CEP engine to use. We ended up not using it as its performance wasn't on par with other offerings. I hope the performance improvements they've done carry over to the spark streaming product. http://spark.apache.org/streaming/ We ended up using esper and a proprietary engine. The biggest problem with some other streaming utilities is 1) performance 2) you can't step time when doing back testing. ie when replaying a day of trading you'll often have signals that say something like be passive for the next 10 seconds and then go to the bid for 10 seconds and then mid point and if you still aren't done then cross the spread after an additional 10 seconds. When backtesting you obviously don't play back in real time or it would take 6.5 hours to re simulate the day, you play back as fast as you can so you need your CEP system to step time as it goes so it properly fires your time based triggers. I'm checking out memSql right now to see how it lets you step time whit queries. Startup idea.... It would be nice to have one unified way of doing real time and batch processing of data. That way your real time trading engine can be back tested in the same way it runs in production. I think/hope Spark is on the way to solving this. If anyone has any input on the best way to unify streaming vs batch event processing please let me know!!!
Posted by chollida1 on Sep 12, 2014, 7:15 PM
Tags
Related issues
- Not sure what you suggest to do.
- Why is this not yet published on the official website and why is there no CVE?
- The biggest problem with having an LLM do this, from what I've observed of at least frontier models, is that the way they are trained...
- The biggest problem with MOND is that it was hand-crafted with additional parameters simply to fit the data.
- (My comment is about detached houses).
- The biggest problem with this is that I can't run my company's own internal benchmarks against this API, because they will be trained int...
Raise real-world problems, find people facing the same issue, and discuss solutions.9,000 issues0 comments
hackernews (Other)
Not sure what you suggest to do.
NewWhy is this not yet published on the official website and why is there no CVE?
NewThe biggest problem with having an LLM do this, from what I've observed of at least frontier models, is that the way they are trained...
The biggest problem with MOND is that it was hand-crafted with additional parameters simply to fit the data.
(My comment is about detached houses).
The biggest problem with this is that I can't run my company's own internal benchmarks against this API, because they will be trained int...
> steer Claude away from over-engineering parts of the system in response to hypothetical concerns.
Why is there no comparison with FLUX 3 Image?
Great to see this: > This release adds significant internal changes to Spark focused on improving performance for large scale workloads.
Great to see this: > This release adds significant internal changes to Spark focused on improving performance for large scale workloads. We looked at Spark Streaming briefly when choosing which CEP engine to use. We ended up not using it as its performance wasn't on par with other offerings. I hope the performance improvements they've done carry over to the spark streaming product. http://spark.apache.org/streaming/ We ended up using esper and a proprietary engine. The biggest problem with some other streaming utilities is 1) performance 2) you can't step time when doing back testing. ie when replaying a day of trading you'll often have signals that say something like be passive for the next 10 seconds and then go to the bid for 10 seconds and then mid point and if you still aren't done then cross the spread after an additional 10 seconds. When backtesting you obviously don't play back in real time or it would take 6.5 hours to re simulate the day, you play back as fast as you can so you need your CEP system to step time as it goes so it properly fires your time based triggers. I'm checking out memSql right now to see how it lets you step time whit queries. Startup idea.... It would be nice to have one unified way of doing real time and batch processing of data. That way your real time trading engine can be back tested in the same way it runs in production. I think/hope Spark is on the way to solving this. If anyone has any input on the best way to unify streaming vs batch event processing please let me know!!!
Comments (0)
No comments yet. Start the discussion.