c# - Way to read (or edit) big JSON from / to stream

Question

(Answered yet - at least 3 solutions left there instead of original question.)
I was been trying to parse & split big JSON, but did not want to modify content.
Floating point conversions changed numbers till FloatParseHandling change.

Similar loop can split 1/4GB JSON on my machine in 40s using only 14MB of RAM comparing to 30s/5-7GB using common Stream.ReadToEnd -> out of or exhaust free RAM -> crash or "stop" approach.

Wanted also to verify results by binary comparision then, but lot of numbers changed.

jsonReader.FloatParseHandling = FloatParseHandling.Decimal;

using Newtonsoft.Json; // intentionally ugly - complete working code

long batchSize = 500000, start = 0, end = 0, pos = 0;
bool neverEnd = true;
while (neverEnd) {
  end = start + batchSize - 1;
  var sr = new StreamReader(File.Open("bigOne.json", FileMode.Open, FileAccess.Read));
  var sw = new StreamWriter(new FileStream(@"PartNo" + start + ".json", FileMode.Create));
  using (JsonWriter writer = new JsonTextWriter(sw))
  using (var jsonR = new JsonTextReader(sr)) {
    jsonR.FloatParseHandling = FloatParseHandling.Decimal;
    while (neverEnd) {
      neverEnd &= jsonR.Read();
      if (jsonR.TokenType == JsonToken.StartObject
       && jsonR.Path.IndexOf("BigArrayPathStart") == 0) { // batters[0] ... batters[3]
        if (pos > end) break;
        if (pos++ < start) {
          do { jsonR.Read(); } while (jsonR.TokenType != JsonToken.EndObject);
          continue;
        }
      }

      if (jsonR.TokenType >= JsonToken.PropertyName){ writer.WriteToken(jsonR); }
      else if (jsonR.TokenType == JsonToken.StartObject) { writer.WriteStartObject(); }
      else if (jsonR.TokenType == JsonToken.StartArray) { writer.WriteStartArray(); }
      else if (jsonR.TokenType == JsonToken.StartConstructor) {
          writer.WriteStartConstructor(jsonR.Value.ToString());
      }
    }
    start = pos; pos = 0;
  }
}

Answer 1

Gason translated to C# is probably quickest parser in C# language now, speed similar to C++ version (Debug Build, 2x slower in Release), memory consumption 2x bigger: https://github.com/eltomjan/gason

(Disclaimer: I am affiliated with this C# fork of Gason.)

Parser has experimental feature - exit after parsing predefined # of lines in last array and next time continue after last item with next batch:

using Gason;

int endPos = -1;
JsonValue jsn;
Byte[] raw;

String json = @"{""id"":""0001"",""type"":""donut"",""name"":""Cake"",""ppu"":0.55, 
  ""batters"": [ { ""id"": ""1001"", ""type"": ""Regular"" },
                 { ""id"": ""1002"", ""type"": ""Chocolate"" },
                 { ""id"": ""1003"", ""type"": ""Blueberry"" }, 
                 { ""id"": ""1004"", ""type"": ""Devil's Food"" } ]
  }"
raw = Encoding.UTF8.GetBytes(json);
ByteString[] keys = new ByteString[]
{
    new ByteString("batters"),
    null
};
Parser jsonParser = new Parser(true); // FloatAsDecimal (,JSON stack array size=32)
jsonParser.Parse(raw, ref endPos, out jsn, keys, 2, 0, 2); // batters / null path...
ValueWriter wr = new ValueWriter(); // read only 1st 2
using (StreamWriter sw = new StreamWriter(Console.OpenStandardOutput()))
{
    sw.AutoFlush = true;
    wr.DumpValueIterative(sw, jsn, raw);
}
Parser.Parse(raw, ref endPos, out jsn, keys, 2, endPos, 2); // and now following 2
using (StreamWriter sw = new StreamWriter(Console.OpenStandardOutput()))
{
    sw.AutoFlush = true;
    wr.DumpValueIterative(sw, jsn, raw);
}

It is a quick and simple option to split long JSONs now - whole 1/4GB, <18Mio rows in main array in <5,3s on a quick machine (Debug Build) using <950MB RAM, Newtonsoft.Json consumed >30s/5.36GB. If parsing only first 100 rows <330ms, >250MB RAM.
In Release Build even better <3.2s where Newton spent >29.3s (>10.8x better performance).

1st Parse:
{
  "id": "0001",
  "type": "donut",
  "name": "Cake",
  "ppu": 0.55,
  "batters": [
    {
      "id": "1001",
      "type": "Regular"
    },
    {
      "id": "1002",
      "type": "Chocolate"
    }
  ]
}
2nd Parse:
{
  "id": "0001",
  "type": "donut",
  "name": "Cake",
  "ppu": 0.55,
  "batters": [
    {
      "id": "1003",
      "type": "Blueberry"
    },
    {
      "id": "1004",
      "type": "Devil's Food"
    }
  ]
}