[{"data":1,"prerenderedAt":551},["ShallowReactive",2],{"lesson-2026-04-04":3},{"id":4,"title":5,"body":6,"date":534,"description":535,"difficulty":536,"extension":537,"format":538,"meta":539,"navigation":440,"path":540,"progression":68,"seo":541,"stem":542,"subtopic":543,"tags":544,"tldr":547,"topic":548,"triggered":549,"__hash__":550},"lessons\u002Flessons\u002F2026-04-04.md","jq's Transformation Engine: Advanced JSON Manipulation",{"type":7,"value":8,"toc":523},"minimark",[9,13,32,37,45,48,175,178,194,197,201,208,211,223,230,234,241,244,266,269,273,280,297,300,304,315,322,326,333,336,351,358,362,375,393,396,400,414,418,421,516,519],[10,11,5],"h1",{"id":12},"jqs-transformation-engine-advanced-json-manipulation",[14,15,16,17,21,22,25,26,31],"p",{},"Most developers know jq for basic field extraction: ",[18,19,20],"code",{},"jq '.name'"," or ",[18,23,24],{},"jq '.items[0]'",". But ",[27,28,30],"cite",{"index":29},"2-6,6-28","jq can mangle the data format that you have into the one that you want with very little effort",", and its real power lies in sophisticated transformations that reshape entire data structures.",[33,34,36],"h2",{"id":35},"beyond-basic-filters-the-art-of-reconstruction","Beyond Basic Filters: The Art of Reconstruction",[14,38,39,40,44],{},"The key insight is that ",[27,41,43],{"index":42},"7-32,7-33","every filter has an input and an output. Even literals like \"hello\" or 42 are filters - they take an input but always produce the same literal as output",". This means you can construct entirely new JSON structures by combining filters creatively.",[14,46,47],{},"Consider this common scenario: you have an API response with nested user data, but you need a flat structure for a CSV export:",[49,50,55],"pre",{"className":51,"code":52,"language":53,"meta":54,"style":54},"language-json shiki shiki-themes github-light","{\n  \"users\": [\n    {\n      \"id\": 1,\n      \"profile\": {\"name\": \"Alice\", \"email\": \"alice@example.com\"},\n      \"metadata\": {\"created\": \"2023-01-15\", \"role\": \"admin\"}\n    }\n  ]\n}\n","json","",[18,56,57,66,76,82,97,129,158,164,170],{"__ignoreMap":54},[58,59,62],"span",{"class":60,"line":61},"line",1,[58,63,65],{"class":64},"sgsFI","{\n",[58,67,69,73],{"class":60,"line":68},2,[58,70,72],{"class":71},"sYu0t","  \"users\"",[58,74,75],{"class":64},": [\n",[58,77,79],{"class":60,"line":78},3,[58,80,81],{"class":64},"    {\n",[58,83,85,88,91,94],{"class":60,"line":84},4,[58,86,87],{"class":71},"      \"id\"",[58,89,90],{"class":64},": ",[58,92,93],{"class":71},"1",[58,95,96],{"class":64},",\n",[58,98,100,103,106,109,111,115,118,121,123,126],{"class":60,"line":99},5,[58,101,102],{"class":71},"      \"profile\"",[58,104,105],{"class":64},": {",[58,107,108],{"class":71},"\"name\"",[58,110,90],{"class":64},[58,112,114],{"class":113},"sYBdl","\"Alice\"",[58,116,117],{"class":64},", ",[58,119,120],{"class":71},"\"email\"",[58,122,90],{"class":64},[58,124,125],{"class":113},"\"alice@example.com\"",[58,127,128],{"class":64},"},\n",[58,130,132,135,137,140,142,145,147,150,152,155],{"class":60,"line":131},6,[58,133,134],{"class":71},"      \"metadata\"",[58,136,105],{"class":64},[58,138,139],{"class":71},"\"created\"",[58,141,90],{"class":64},[58,143,144],{"class":113},"\"2023-01-15\"",[58,146,117],{"class":64},[58,148,149],{"class":71},"\"role\"",[58,151,90],{"class":64},[58,153,154],{"class":113},"\"admin\"",[58,156,157],{"class":64},"}\n",[58,159,161],{"class":60,"line":160},7,[58,162,163],{"class":64},"    }\n",[58,165,167],{"class":60,"line":166},8,[58,168,169],{"class":64},"  ]\n",[58,171,173],{"class":60,"line":172},9,[58,174,157],{"class":64},[14,176,177],{},"The transformation becomes:",[49,179,183],{"className":180,"code":181,"language":182,"meta":54,"style":54},"language-bash shiki shiki-themes github-light","jq '.users[] | {id, name: .profile.name, email: .profile.email, role: .metadata.role}'\n","bash",[18,184,185],{"__ignoreMap":54},[58,186,187,191],{"class":60,"line":61},[58,188,190],{"class":189},"s7eDp","jq",[58,192,193],{"class":113}," '.users[] | {id, name: .profile.name, email: .profile.email, role: .metadata.role}'\n",[14,195,196],{},"But this is still basic. The real power emerges when you start thinking in terms of data flows and transformations.",[33,198,200],{"id":199},"recursive-descent-navigating-complex-structures","Recursive Descent: Navigating Complex Structures",[14,202,203,207],{},[27,204,206],{"index":205},"7-16","The recurse(f) function allows you to search through a recursive structure, and extract interesting data from all levels",". This becomes invaluable when dealing with nested configuration files, file system representations, or organizational hierarchies.",[14,209,210],{},"Suppose you have a complex configuration object and need to find all database connection strings, regardless of nesting level:",[49,212,214],{"className":180,"code":213,"language":182,"meta":54,"style":54},"jq 'recurse | objects | select(has(\"connection_string\")) | .connection_string'\n",[18,215,216],{"__ignoreMap":54},[58,217,218,220],{"class":60,"line":61},[58,219,190],{"class":189},[58,221,222],{"class":113}," 'recurse | objects | select(has(\"connection_string\")) | .connection_string'\n",[14,224,225,226,229],{},"The beauty of ",[18,227,228],{},"recurse"," is that it generates all possible paths through your data structure. You can then filter and transform at any level.",[33,231,233],{"id":232},"variable-binding-breaking-complex-transformations","Variable Binding: Breaking Complex Transformations",[14,235,236,240],{},[27,237,239],{"index":238},"3-15,3-17,3-18","Variables are an absolute necessity in most programming languages, but they're relegated to an \"advanced feature\" in jq. If you calculate a value, and you want to use it more than once, you'll need to store it in a variable",".",[14,242,243],{},"Here's where jq variables become crucial for complex transformations:",[49,245,247],{"className":180,"code":246,"language":182,"meta":54,"style":54},"jq '.items[] as $item | \n    ($item.price * $item.quantity) as $total |\n    {name: $item.name, unit_price: $item.price, total: $total, tax: ($total * 0.08)}'\n",[18,248,249,256,261],{"__ignoreMap":54},[58,250,251,253],{"class":60,"line":61},[58,252,190],{"class":189},[58,254,255],{"class":113}," '.items[] as $item | \n",[58,257,258],{"class":60,"line":68},[58,259,260],{"class":113},"    ($item.price * $item.quantity) as $total |\n",[58,262,263],{"class":60,"line":78},[58,264,265],{"class":113},"    {name: $item.name, unit_price: $item.price, total: $total, tax: ($total * 0.08)}'\n",[14,267,268],{},"This pattern—binding intermediate results to variables—prevents redundant calculations and makes complex transformations readable.",[33,270,272],{"id":271},"function-definition-building-reusable-logic","Function Definition: Building Reusable Logic",[14,274,275,279],{},[27,276,278],{"index":277},"3-19","It is also possible to define functions in jq, although this is is a feature whose biggest use is defining jq's standard library",". But you can define your own functions within filters:",[49,281,283],{"className":180,"code":282,"language":182,"meta":54,"style":54},"jq 'def calculate_discount(rate): . * (1 - rate);\n    .products[] | {name, original: .price, discounted: (.price | calculate_discount(0.15))}'\n",[18,284,285,292],{"__ignoreMap":54},[58,286,287,289],{"class":60,"line":61},[58,288,190],{"class":189},[58,290,291],{"class":113}," 'def calculate_discount(rate): . * (1 - rate);\n",[58,293,294],{"class":60,"line":68},[58,295,296],{"class":113},"    .products[] | {name, original: .price, discounted: (.price | calculate_discount(0.15))}'\n",[14,298,299],{},"Functions become essential when you're applying the same transformation logic across multiple data points or building up complex calculations step by step.",[33,301,303],{"id":302},"stream-processing-handling-large-data-sets","Stream Processing: Handling Large Data Sets",[14,305,306,310,311,314],{},[27,307,309],{"index":308},"3-8,3-36","Instead of running the filter for each JSON object in the input, read the entire input stream into a large array and run the filter just once"," with ",[18,312,313],{},"--slurp",", but sometimes you want the opposite: process data as a stream to handle large files that won't fit in memory.",[14,316,317,321],{},[27,318,320],{"index":319},"7-1,7-36","splits(regex), splits(regex; flags) These provide the same results as their split counterparts, but as a stream instead of an array",". This streaming approach becomes critical when processing large log files or data exports.",[33,323,325],{"id":324},"pattern-matching-and-text-processing","Pattern Matching and Text Processing",[14,327,328,329,240],{},"Modern jq includes powerful regex capabilities. ",[27,330,332],{"index":331},"7-2,7-5","sub(regex; tostring) and gsub is like sub but all the non-overlapping occurrences of the regex are replaced by the string, after interpolation",[14,334,335],{},"For log processing, you might extract structured data from unstructured text:",[49,337,339],{"className":180,"code":338,"language":182,"meta":54,"style":54},"jq -R 'capture(\"(?\u003Ctimestamp>\\\\d{4}-\\\\d{2}-\\\\d{2}) (?\u003Clevel>\\\\w+) (?\u003Cmessage>.*)\") | select(.level == \"ERROR\")'\n",[18,340,341],{"__ignoreMap":54},[58,342,343,345,348],{"class":60,"line":61},[58,344,190],{"class":189},[58,346,347],{"class":71}," -R",[58,349,350],{"class":113}," 'capture(\"(?\u003Ctimestamp>\\\\d{4}-\\\\d{2}-\\\\d{2}) (?\u003Clevel>\\\\w+) (?\u003Cmessage>.*)\") | select(.level == \"ERROR\")'\n",[14,352,353,354,357],{},"The ",[18,355,356],{},"-R"," flag treats each line as a string rather than JSON, perfect for processing logs or other text data.",[33,359,361],{"id":360},"advanced-array-operations","Advanced Array Operations",[14,363,364,374],{},[27,365,367,368,373],{"index":366},"6-25,6-27","jq 'sort_by(.price)' and jq '",[58,369,240,370,372],{},[58,371],{}," .price"," | add'"," show basic array operations, but you can build sophisticated aggregations:",[49,376,378],{"className":180,"code":377,"language":182,"meta":54,"style":54},"# Group by category, then sum prices within each group\njq 'group_by(.category) | map({category: .[0].category, total: map(.price) | add, count: length})'\n",[18,379,380,386],{"__ignoreMap":54},[58,381,382],{"class":60,"line":61},[58,383,385],{"class":384},"sAwPA","# Group by category, then sum prices within each group\n",[58,387,388,390],{"class":60,"line":68},[58,389,190],{"class":189},[58,391,392],{"class":113}," 'group_by(.category) | map({category: .[0].category, total: map(.price) | add, count: length})'\n",[14,394,395],{},"This pattern—group, map over groups, aggregate within groups—handles most complex reporting scenarios.",[33,397,399],{"id":398},"pro-tip","Pro Tip",[14,401,402,403,406,407,410,411,240],{},"When building complex jq transformations, develop them incrementally. Start with the basic structure, then add one transformation at a time. Use ",[18,404,405],{},"jq -n 'null'"," to test expressions without input data, and remember that you can pipe intermediate results to ",[18,408,409],{},"debug"," to inspect the data flow: ",[18,412,413],{},"jq '.items[] | debug | .price'",[33,415,417],{"id":416},"example","Example",[14,419,420],{},"Here's a real-world transformation that converts a messy API response into a clean summary report:",[49,422,424],{"className":180,"code":423,"language":182,"meta":54,"style":54},"# Input: complex e-commerce API response\n# Output: clean sales summary by region\n\njq '\n  def safe_divide(a; b): if b == 0 then 0 else a\u002Fb end;\n  \n  .orders[]\n  | group_by(.shipping.region)\n  | map({\n      region: .[0].shipping.region,\n      orders: length,\n      revenue: map(.items[].price * .items[].quantity) | add,\n      avg_order: safe_divide(map(.items[].price * .items[].quantity) | add; length)\n    })\n  | sort_by(-.revenue)\n'\n",[18,425,426,431,436,442,449,454,459,464,469,474,480,486,492,498,504,510],{"__ignoreMap":54},[58,427,428],{"class":60,"line":61},[58,429,430],{"class":384},"# Input: complex e-commerce API response\n",[58,432,433],{"class":60,"line":68},[58,434,435],{"class":384},"# Output: clean sales summary by region\n",[58,437,438],{"class":60,"line":78},[58,439,441],{"emptyLinePlaceholder":440},true,"\n",[58,443,444,446],{"class":60,"line":84},[58,445,190],{"class":189},[58,447,448],{"class":113}," '\n",[58,450,451],{"class":60,"line":99},[58,452,453],{"class":113},"  def safe_divide(a; b): if b == 0 then 0 else a\u002Fb end;\n",[58,455,456],{"class":60,"line":131},[58,457,458],{"class":113},"  \n",[58,460,461],{"class":60,"line":160},[58,462,463],{"class":113},"  .orders[]\n",[58,465,466],{"class":60,"line":166},[58,467,468],{"class":113},"  | group_by(.shipping.region)\n",[58,470,471],{"class":60,"line":172},[58,472,473],{"class":113},"  | map({\n",[58,475,477],{"class":60,"line":476},10,[58,478,479],{"class":113},"      region: .[0].shipping.region,\n",[58,481,483],{"class":60,"line":482},11,[58,484,485],{"class":113},"      orders: length,\n",[58,487,489],{"class":60,"line":488},12,[58,490,491],{"class":113},"      revenue: map(.items[].price * .items[].quantity) | add,\n",[58,493,495],{"class":60,"line":494},13,[58,496,497],{"class":113},"      avg_order: safe_divide(map(.items[].price * .items[].quantity) | add; length)\n",[58,499,501],{"class":60,"line":500},14,[58,502,503],{"class":113},"    })\n",[58,505,507],{"class":60,"line":506},15,[58,508,509],{"class":113},"  | sort_by(-.revenue)\n",[58,511,513],{"class":60,"line":512},16,[58,514,515],{"class":113},"'\n",[14,517,518],{},"This single jq expression extracts orders, groups by region, calculates totals and averages, and sorts by revenue—replacing dozens of lines of traditional scripting with a declarative transformation.",[520,521,522],"style",{},"html pre.shiki code .s7eDp, html code.shiki .s7eDp{--shiki-default:#6F42C1}html pre.shiki code .sYBdl, html code.shiki .sYBdl{--shiki-default:#032F62}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html pre.shiki code .sYu0t, html code.shiki .sYu0t{--shiki-default:#005CC5}html pre.shiki code .sAwPA, html code.shiki .sAwPA{--shiki-default:#6A737D}html pre.shiki code .sgsFI, html code.shiki .sgsFI{--shiki-default:#24292E}",{"title":54,"searchDepth":68,"depth":68,"links":524},[525,526,527,528,529,530,531,532,533],{"id":35,"depth":68,"text":36},{"id":199,"depth":68,"text":200},{"id":232,"depth":68,"text":233},{"id":271,"depth":68,"text":272},{"id":302,"depth":68,"text":303},{"id":324,"depth":68,"text":325},{"id":360,"depth":68,"text":361},{"id":398,"depth":68,"text":399},{"id":416,"depth":68,"text":417},"2026-04-04","Most developers know jq for basic field extraction: jq '.name' or jq '.items[0]'. But jq can mangle the data format that you have into the one that you want with very little effort, and its real power lies in sophisticated transformations that reshape entire data structures.","intermediate","md","deep-dive",{},"\u002Flessons\u002F2026-04-04",{"title":5,"description":535},"lessons\u002F2026-04-04","jq-transformations",[190,53,545,546],"data-transformation","cli","Master jq's advanced transformation patterns including recursive operations, custom functions, and complex data reshaping that turn unwieldy JSON into exactly what you need.","tools",false,"ZyCM16yTYgnJlIoUVBA9_ngfZyUx6HRQCBsWASZTxKc",1775506334963]