Variant Data Type Support
### Proposed Change We would like to propose to add Variant type to Iceberg data types. Variant data types allow for the efficient binary encoding of dynamic semi-structured data such as JSON, Avro,Parquet, etc. By encoding semi-structured data as a variant column, we retain the flexibility of the source data, while allowing query engines to more efficiently operate on the data. With the support of Variant type, such data can be encoded in an efficient binary representation internally for better performance. Without that, we need to parse the data in its format inefficiently. This will allow the following use cases: - Create an Iceberg table with a Variant column `CREATE OR REPLACE TABLE car_sales(record Variant);` - Insert semi-structured data into the Variant column `INSERT INTO car_sales SELECT PARSE_JSON(<json_string>)` - Query against the semi-structured data `SELECT VARIANT_GET(record, '$.dealer.ship', 'string') FROM car_sales` ### Proposal document https://docs.google.com/document/d/1sq70XDiWJ2DemWyA5dVB80gKzwi0CWoM0LOWM7VJVd8/edit?tab=t.0 ### Specifications - [X] Table - [ ] View - [ ] REST - [ ] Puffin - [ ] Encryption - [ ] Other
Posted by sfc-gh-aixu on May 29, 2024, 12:12 AM
Tags
Related issues
- some bike vendor told me that the biggest problem with regular to ebike conversion was that regular bike frames were not strong enough to...
- I think of this as the kiosk economy.
- What a goofy comment.
- There's lots of evidence for positive influences of Caffeine on the body (some negative as well), especially the brain.
- Compose support in WSLc
- [RFC] Roadmap to v2.0
Raise real-world problems, find people facing the same issue, and discuss solutions.2,000 issues0 comments
support (Other)
some bike vendor told me that the biggest problem with regular to ebike conversion was that regular bike frames were not strong enough to...
I think of this as the kiosk economy.
What a goofy comment.
There's lots of evidence for positive influences of Caffeine on the body (some negative as well), especially the brain.
Compose support in WSLc
[RFC] Roadmap to v2.0
[Feature]: Add TurboQuant Support for KV Cache Quantization
Support for AGENTS.md and .agents/skills/, the community has been asking since August 2025
Variant Data Type Support
### Proposed Change We would like to propose to add Variant type to Iceberg data types. Variant data types allow for the efficient binary encoding of dynamic semi-structured data such as JSON, Avro,Parquet, etc. By encoding semi-structured data as a variant column, we retain the flexibility of the source data, while allowing query engines to more efficiently operate on the data. With the support of Variant type, such data can be encoded in an efficient binary representation internally for better performance. Without that, we need to parse the data in its format inefficiently. This will allow the following use cases: - Create an Iceberg table with a Variant column `CREATE OR REPLACE TABLE car_sales(record Variant);` - Insert semi-structured data into the Variant column `INSERT INTO car_sales SELECT PARSE_JSON(<json_string>)` - Query against the semi-structured data `SELECT VARIANT_GET(record, '$.dealer.ship', 'string') FROM car_sales` ### Proposal document https://docs.google.com/document/d/1sq70XDiWJ2DemWyA5dVB80gKzwi0CWoM0LOWM7VJVd8/edit?tab=t.0 ### Specifications - [X] Table - [ ] View - [ ] REST - [ ] Puffin - [ ] Encryption - [ ] Other
Comments (0)
No comments yet. Start the discussion.