It isn't trivial actually.
It isn't trivial actually. Detection would be a multipass process, the first pass would be a measurement of entropy, which would also flag compressed data (which would be very common). The second pass being magic number detection (file headers, can't be trusted). The third pass would likely be some sort of conformance test to every know compression format. The result of all this io intensive processing would not be a definitive answer. If you want to get a taste of how difficult it would be to implement some method of detection, with a useful detection rate at scale, try to write some software that can detect localization encoding of strings on a binary stream. There is a reason why automatic translation of unknown localization doesn't really exist, there have been plenty of attempts, but nothing that results in a definitive answer. I once wrote some software that would parse very poorly written csv files, it was one of the most frustrating projects I've ever done - and taught me the value of standards conformance. Basically, if the data source is non-cooperative, you aren't going to be able to reliably use the data.
Posted by woodman on Jun 5, 2015, 12:06 AM
Tags
Related issues
- Some good points here, and I definitely hate the "unalive" movement but like someone else said, it has gone beyond self-censorship to in-...
- It’s hard to take the article seriously when there’s no real discussion about how consciousness is defined and it takes Anthropic’s views...
- you know what i hate?
- > I honestly do not understand why any average person would buy anything else than a macbook today.
- Codex App frequently freezes/stutters on Windows 11 Pro despite sufficient system resources
- Add "Basic" and "Standard" plans
Raise real-world problems, find people facing the same issue, and discuss solutions.6,000 issues0 comments
ux (SaaS)
Some good points here, and I definitely hate the "unalive" movement but like someone else said, it has gone beyond self-censorship to in-...
It’s hard to take the article seriously when there’s no real discussion about how consciousness is defined and it takes Anthropic’s views...
you know what i hate?
> I honestly do not understand why any average person would buy anything else than a macbook today.
Codex App frequently freezes/stutters on Windows 11 Pro despite sufficient system resources
Add "Basic" and "Standard" plans
FEATURE: Multi-window support in Claude Code Desktop
Hi HN I built CookPal to solve a small but annoying problem for myself: I kept saving recipes from websites, TikTok, YouTube, and Instag...
It isn't trivial actually.
It isn't trivial actually. Detection would be a multipass process, the first pass would be a measurement of entropy, which would also flag compressed data (which would be very common). The second pass being magic number detection (file headers, can't be trusted). The third pass would likely be some sort of conformance test to every know compression format. The result of all this io intensive processing would not be a definitive answer. If you want to get a taste of how difficult it would be to implement some method of detection, with a useful detection rate at scale, try to write some software that can detect localization encoding of strings on a binary stream. There is a reason why automatic translation of unknown localization doesn't really exist, there have been plenty of attempts, but nothing that results in a definitive answer. I once wrote some software that would parse very poorly written csv files, it was one of the most frustrating projects I've ever done - and taught me the value of standards conformance. Basically, if the data source is non-cooperative, you aren't going to be able to reliably use the data.
Comments (0)
No comments yet. Start the discussion.