From 48b1c2b2c2f2e6dce303a9a0a14d757746a841a0 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 18 Oct 2026 11:02:57 +0000 Subject: [PATCH 01/10] filters: implement camelize, underscorize, escapecpp and toyaml, remove toxml (0233) The five C++-only filters were registered but rendered an empty string. Per docs/release-2.0.md D16, camelize(upper_first=true), underscorize, escapecpp and toyaml now render the output docs/cpp-only-filters.md describes; toxml is removed and is an unknown filter, as in Jinja2. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_017KNXpdwgCcK4gqUkYrVXzK --- docs/api-2.0.md | 8 + docs/cpp-only-filters.md | 82 ++++++ docs/parity.md | 6 +- docs/tasks/0233-cpp-only-filter-stubs.md | 10 +- src/filter_factories.cpp | 1 - src/filters.h | 1 - src/serialize_filters.cpp | 341 ++++++++++++++++++++++- src/string_converter_filter.cpp | 182 ++++++++++++ test/filters_test.cpp | 70 +++++ test/parity/cases/filters.py | 2 + test/parity/expected/filters.json | 70 +++++ 11 files changed, 765 insertions(+), 8 deletions(-) create mode 100644 docs/cpp-only-filters.md diff --git a/docs/api-2.0.md b/docs/api-2.0.md index 55decd1e..fdcd83cf 100644 --- a/docs/api-2.0.md +++ b/docs/api-2.0.md @@ -418,6 +418,14 @@ inside an `if` statement or an inline if, where it fails when it runs (docs/task 1.x let a `UserCallable` passed in the render parameters or the globals act as a filter or test, found when it ran; `Settings::userCallablesAsFilters = true` keeps that behaviour. +### 5.7 C++-only filters render, `toxml` is gone (release note) + +In 1.x `camelize`, `underscorize`, `escapecpp`, `toxml` and `toyaml` were registered but +rendered an empty string. In 2.0 the first four render the output that +[cpp-only-filters.md](cpp-only-filters.md) describes, and `toxml` is removed: a template that +uses it fails `Template::Load` with `ErrorCode::UnknownFilter` (docs/tasks/0233, +docs/release-2.0.md D16). Templates that leaned on the empty output drop the filter. + ## 6. Release options | Option | What ships | Users get | Cost | diff --git a/docs/cpp-only-filters.md b/docs/cpp-only-filters.md new file mode 100644 index 00000000..fb976e27 --- /dev/null +++ b/docs/cpp-only-filters.md @@ -0,0 +1,82 @@ +# C++-only filters + +Jinja2C++ adds a few filters that Python Jinja2 does not have. A template that uses them +does not load in Jinja2 (`No filter named ...`). Since 2.0 each has the output described +here (docs/tasks/0233); before, `camelize`, `underscorize`, `escapecpp`, `toxml` and +`toyaml` were registered but rendered an empty string. + +| Filter | Output | +|---|---| +| `applymacro(macro=name, ...)` | The value passed to the macro `name`, with the other arguments | +| `camelize(upper_first=true)` | An identifier in CamelCase | +| `underscorize` | An identifier in snake_case | +| `escapecpp` | The text escaped for a C++ string literal | +| `toyaml` | The value as a block-style YAML document | + +`toxml` was removed in 2.0: JSON-like data has no single XML mapping, so it would be a format +of our own to define and support (docs/release-2.0.md, D16). A template that uses it fails +to load with `ErrorCode::UnknownFilter`; `xmlattr` covers attribute lists, and a macro +can write any XML shape. + +## `camelize` and `underscorize` + +Both take the value's `str()` and split it into words: runs of letters and digits (Unicode +`\w` without `_`), split again before an uppercase letter that follows a lowercase letter +or a digit (`fooBar`, `Vector3D`) and before the last of a run of uppercase letters that a +lowercase one follows (`HTTPServer` is `HTTP` and `Server`). Everything between words +(`_`, `-`, spaces, punctuation) is dropped. + +- `camelize` gives each word a titlecase first letter and lowercase others and joins them: + `device_type`, `device-type` and `DEVICE_TYPE` are `DeviceType`, `HTTPServer` is + `HttpServer`. With `upper_first=false` the first word is all lowercase: `deviceType`, + `httpServer`. +- `underscorize` joins the words in lowercase with `_`: `DeviceType` is `device_type`, + `getHTTPResponseCode` is `get_http_response_code`, `__init__` is `init`. + +Case mapping is Unicode's, as in `upper`/`lower`: `straße_name|camelize` is `StraßeName`. + +## `escapecpp` + +The value's `str()` as the contents of a C++ string literal, without the quotes: `\` and +`"` get a backslash, `\n`, `\r`, `\t`, `\a`, `\b`, `\f` and `\v` their escapes, and other +control characters (below U+0020, and U+007F) three octal digits (`\001`), which cannot run +into a digit that follows the way `\x` does. Other characters, `'` and non-ASCII ones +included, are kept, so a UTF-8 template gives a literal for a UTF-8 source file. + +``` +const char* text = "{{ message|escapecpp }}"; +``` + +## `toyaml` + +A YAML document that YAML 1.1 and 1.2 loaders (PyYAML, yaml-cpp, ...) read back as the same +data, with no trailing newline: + +- `none` (and a plain undefined) is `null`, booleans `true`/`false`, integers in decimal, + floats in Python's shortest form with a `.` always present (`1.0e+20`), `.nan`, `.inf` + and `-.inf`. +- A mapping is `key: value` lines in the mapping's order; a list is `- item` lines; a + `dictsort` or `items` pair is a two-item list. Nested collections are indented by two + spaces, a collection inside a list starts on the item's line, and an empty one is `[]` + or `{}`. +- A string (a key too) is written plain only when it cannot be read as anything else: it + starts with a letter, `_`, `/` or a printable non-ASCII character, holds only those, + digits, spaces and `-.()+`, does not end with a space and is not a null or boolean word + (`y`, `n`, `yes`, `no`, `on`, `off`, `true`, `false`, `null`, in any case). Any other + string is double-quoted with YAML escapes (`\"`, `\\`, `\n`, `\t`, `\xHH`, `\uHHHH`, ...). +- `namespace()`, `cycler()`, `joiner()`, macros and strict undefined values raise an error. + +``` +{{ {'name': 'demo', 'ports': [80, 443], 'tls': {'enabled': true}}|toyaml }} +``` + +```yaml +name: demo +ports: + - 80 + - 443 +tls: + enabled: true +``` + +The result is an ordinary string: under autoescape it is HTML-escaped like any other. diff --git a/docs/parity.md b/docs/parity.md index d85efadc..6af924b3 100644 --- a/docs/parity.md +++ b/docs/parity.md @@ -255,8 +255,10 @@ storing a container in itself raises instead of printing `[...]` (deliberate, 00 | `xmlattr` | ✅ | `xmlattr*` | | | Unknown filter is an error; in a branch never taken it is not | ✅ | `unknown_filter*` | | -C++-only filters (`camelize`, `underscorize`, `escapecpp`, `toxml`, `toyaml`, -`applymacro`) and the `startsWith` test are deliberate extensions; Jinja2 rejects them. +C++-only filters (`camelize`, `underscorize`, `escapecpp`, `toyaml`, `applymacro`; see +[cpp-only-filters.md](cpp-only-filters.md)) and the `startsWith` test are deliberate +extensions; Jinja2 rejects them. `toxml` was removed in 2.0 and is an unknown filter, as in +Jinja2 (0233). ## Statements (`statements`) diff --git a/docs/tasks/0233-cpp-only-filter-stubs.md b/docs/tasks/0233-cpp-only-filter-stubs.md index aec27e31..7ec93233 100644 --- a/docs/tasks/0233-cpp-only-filter-stubs.md +++ b/docs/tasks/0233-cpp-only-filter-stubs.md @@ -1,9 +1,9 @@ --- -status: open +status: done priority: medium area: api issues: ["https://github.com/jinja2cpp/Jinja2Cpp/issues/41"] -touches: [src/filter_factories.cpp, src/string_converter_filter.cpp, src/serialize_filters.cpp] +touches: [src/filter_factories.cpp, src/string_converter_filter.cpp, src/serialize_filters.cpp, src/filters.h, test/filters_test.cpp, docs/cpp-only-filters.md] shares: [docs/parity.md] --- # C++-only filters `camelize`, `underscorize`, `escapecpp`, `toxml`, `toyaml` do nothing @@ -26,3 +26,9 @@ and the `startsWith` test work and stay. **Done when.** Each name either renders its documented output with rows in `test/filters_test.cpp`, or fails Load with `UnknownFilter`; `docs/parity.md` and the 2.0 migration notes (0077) say which. + +**Outcome.** Ruslan chose D16 a (2026-10-18). `camelize(upper_first=true)`, `underscorize`, +`escapecpp` and `toyaml` render the output documented in `docs/cpp-only-filters.md`, with +rows in `test/filters_test.cpp`; `toxml` is removed and fails Load with `UnknownFilter` +(corpus case `filters.unknown_filter_toxml`). Release note: `docs/api-2.0.md` 5.7, for +0077's `MIGRATION.md`. diff --git a/src/filter_factories.cpp b/src/filter_factories.cpp index 3df04841..88496044 100644 --- a/src/filter_factories.cpp +++ b/src/filter_factories.cpp @@ -103,7 +103,6 @@ const std::vector& BuiltinFilters() Entry("sum"), Entry("title"), Entry("tojson"), - Entry("toxml"), Entry("toyaml"), Entry("trim"), Entry("truncate"), diff --git a/src/filters.h b/src/filters.h index dd46467d..6921b9c1 100644 --- a/src/filters.h +++ b/src/filters.h @@ -153,7 +153,6 @@ class Serialize : public FilterBase enum Mode { JsonMode, - XmlMode, YamlMode }; diff --git a/src/serialize_filters.cpp b/src/serialize_filters.cpp index 71bc148a..177c0745 100644 --- a/src/serialize_filters.cpp +++ b/src/serialize_filters.cpp @@ -9,6 +9,7 @@ #include "render_context.h" #include "string_builder.h" #include "undefined.h" +#include "unicode_printable.h" #include "value_visitors.h" #include @@ -183,6 +184,7 @@ Serialize::Serialize(const FilterParams& params, const Serialize::Mode mode) break; } default: + ParseParams(NoArguments(), params); break; } } @@ -467,13 +469,348 @@ class PythonJsonWriter StringBuilder m_out; }; +// toyaml: a YAML document in block style that YAML 1.1 and 1.2 loaders read back as the same +// data. Mappings keep their order, nested collections indent by two spaces, empty ones are +// [] and {}, and the document has no trailing newline. A string is a plain scalar only when +// it cannot be read as anything else: it starts with a letter, _, / or a printable non-ASCII +// character, holds only those, digits, spaces and -.()+, ends with no space and is no +// null/bool word (y, n, yes, no, on, off, true, false, null in any case). Any other string is +// double-quoted with YAML escapes. +class YamlWriter +{ +public: + explicit YamlWriter(RenderContext& context) + : m_context(context) + { + } + + InternalValue Write(const InternalValue& value) + { + WriteNode(value, 0, Position::Document); + return m_out.Build(); + } + +private: + [[noreturn]] void Fail() const + { + m_context.GetRendererCallback()->ThrowRuntimeError(ErrorCode::InvalidValueType, ValuesList{}); + throw std::runtime_error("toyaml(): value is not YAML serializable"); + } + + // The characters of a UTF-8 string as code points, each with its bytes; continuation bytes + // belong to the character before them, as SplitCodePoints splits them + template + static void ForEachChar(std::string_view str, const Fn& fn) + { + size_t pos = 0; + while (pos != str.size()) + { + auto end = pos + 1; + while (end != str.size() && IsCodePointTail(str[end])) + { + ++end; + } + const auto ch = str.substr(pos, end - pos); + uint32_t cp = static_cast(ch[0]); + if (ch.size() > 1) + { + cp &= 0x7FU >> ch.size(); + } + for (size_t n = 1; n < ch.size(); ++n) + { + cp = (cp << 6) | (static_cast(ch[n]) & 0x3F); + } + fn(cp, ch); + pos = end; + } + } + + static bool IsAsciiLetter(uint32_t cp) { return (cp >= 'a' && cp <= 'z') || (cp >= 'A' && cp <= 'Z'); } + + static bool IsPlain(std::string_view str) + { + if (str.empty() || str.back() == ' ') + { + return false; + } + static const char* const words[] = { "y", "n", "yes", "no", "on", "off", "true", "false", "null" }; + for (const auto* word : words) + { + if (str.size() == std::strlen(word) && std::equal(str.begin(), str.end(), word, [](char l, char r) { return (l | 0x20) == r; })) + { + return false; + } + } + bool plain = true; + bool first = true; + ForEachChar(str, [&plain, &first](uint32_t cp, std::string_view) { + const bool startChar = IsAsciiLetter(cp) || cp == '_' || cp == '/' || (cp >= 0x80 && detail::IsPythonPrintable(cp)); + const bool innerChar = (cp >= '0' && cp <= '9') || cp == ' ' || cp == '-' || cp == '.' || cp == '(' || cp == ')' || cp == '+'; + plain = plain && (startChar || (!first && innerChar)); + first = false; + }); + return plain; + } + + void WriteHex(char kind, uint32_t cp, int digits) + { + static const char hexDigits[] = "0123456789ABCDEF"; + m_out.push_back('\\'); + m_out.push_back(kind); + for (int shift = (digits - 1) * 4; shift >= 0; shift -= 4) + { + m_out.push_back(hexDigits[(cp >> shift) & 0xF]); + } + } + + void WriteString(std::string_view str) + { + if (IsPlain(str)) + { + m_out.append(str.data(), str.size()); + return; + } + m_out.push_back('"'); + ForEachChar(str, [this](uint32_t cp, std::string_view ch) { + switch (cp) + { + case '"': + m_out += "\\\""; + return; + case '\\': + m_out += "\\\\"; + return; + case '\0': + m_out += "\\0"; + return; + case '\a': + m_out += "\\a"; + return; + case '\b': + m_out += "\\b"; + return; + case '\t': + m_out += "\\t"; + return; + case '\n': + m_out += "\\n"; + return; + case '\v': + m_out += "\\v"; + return; + case '\f': + m_out += "\\f"; + return; + case '\r': + m_out += "\\r"; + return; + case 0x1B: + m_out += "\\e"; + return; + default: + break; + } + if (cp < 0x20 || cp == 0x7F) + { + WriteHex('x', cp, 2); + } + else if (cp < 0x80 || detail::IsPythonPrintable(cp)) + { + m_out.append(ch.data(), ch.size()); + } + else if (cp < 0x10000) + { + WriteHex('u', cp, 4); + } + else + { + WriteHex('U', cp, 8); + } + }); + m_out.push_back('"'); + } + + // YAML 1.1 reads a float only with a dot in it: 1e+20 is written 1.0e+20 + void WriteFloat(double val) + { + if (std::isnan(val)) + { + m_out += ".nan"; + return; + } + if (std::isinf(val)) + { + m_out += val < 0 ? "-.inf" : ".inf"; + return; + } + const auto text = visitors::PythonFloatText(val); + const auto view = text.View(); + const auto exp = view.find('e'); + if (exp != std::string_view::npos && view.find('.') == std::string_view::npos) + { + m_out += view.substr(0, exp); + m_out += ".0"; + m_out += view.substr(exp); + return; + } + m_out += view; + } + + void NewLine(size_t indent) + { + m_out.push_back('\n'); + for (size_t n = 0; n != indent; ++n) + { + m_out.push_back(' '); + } + } + + // Where a node starts: the document start, after a sequence item's "-" or after a + // mapping key's ":" + enum class Position + { + Document, + SequenceItem, + MappingValue + }; + + // Before a scalar, an empty collection or the first line of a collection: nothing at the + // document start, a space after "-" (a nested collection starts on that line) and, after + // ":", a space or a new line at indent + void StartNode(Position pos, size_t indent, bool blockCollection) + { + if (pos == Position::MappingValue && blockCollection) + { + NewLine(indent); + } + else if (pos != Position::Document) + { + m_out.push_back(' '); + } + } + + // Writes value with its lines at column indent + void WriteNode(const InternalValue& value, size_t indent, Position pos) + { + CheckStack(); + if (const auto* info = GetUndefinedInfo(value)) + { + throw std::runtime_error(std::string("Object of type ") + UndefinedClassName(*info) + " is not YAML serializable"); + } + auto scalar = [this, indent, pos] { StartNode(pos, indent, false); }; + if (value.IsNone() || value.IsUndefined()) + { + scalar(); + m_out += "null"; + } + else if (const auto* b = GetIf(&value)) + { + scalar(); + m_out += *b ? "true" : "false"; + } + else if (const auto* i = GetIf(&value)) + { + scalar(); + m_out += std::to_string(*i); + } + else if (const auto* d = GetIf(&value)) + { + scalar(); + WriteFloat(*d); + } + else if (auto str = GetAsSameString(std::string(), value)) + { + scalar(); + WriteString(*str); + } + else if (const auto* pair = GetIf(&value)) + { + const InternalValueList items{ InternalValue(pair->key), pair->value }; + WriteSequence(items, indent, pos); + } + else if (auto list = AsList(value)) + { + WriteSequence(*list, indent, pos); + } + else if (auto map = AsMap(value)) + { + WriteMapping(*map, indent, pos); + } + else + { + Fail(); + } + } + + template + void WriteSequence(const Items& items, size_t indent, Position pos) + { + bool isFirst = true; + for (const auto& item : items) + { + if (isFirst) + { + StartNode(pos, indent, true); + } + else + { + NewLine(indent); + } + isFirst = false; + m_out.push_back('-'); + WriteNode(item, indent + 2, Position::SequenceItem); + } + if (isFirst) + { + StartNode(pos, indent, false); + m_out += "[]"; + } + } + + void WriteMapping(const MapRef& map, size_t indent, Position pos) + { + if (const char* objectClass = map.GetObjectClass()) + { + // namespace(), cycler() and joiner() are objects, not data + throw std::runtime_error("Object of type "s + objectClass + " is not YAML serializable"); + } + const auto keys = map.GetKeys(); + if (keys.empty()) + { + StartNode(pos, indent, false); + m_out += "{}"; + return; + } + bool isFirst = true; + for (const auto& key : keys) + { + if (isFirst) + { + StartNode(pos, indent, true); + } + else + { + NewLine(indent); + } + isFirst = false; + WriteString(key); + m_out.push_back(':'); + WriteNode(map.GetValueByName(key), indent + 2, Position::MappingValue); + } + } + + RenderContext& m_context; + StringBuilder m_out; +}; + } // namespace InternalValue Serialize::Filter(const InternalValue& value, RenderContext& context) { - if (m_mode != JsonMode) + if (m_mode == YamlMode) { - return InternalValue(); + // YAML is no HTML: under autoescape the result is escaped like any string + return YamlWriter(context).Write(value); } // Jinja2's do_tojson: json.dumps with sort_keys=True, then htmlsafe_json_dumps escapes diff --git a/src/string_converter_filter.cpp b/src/string_converter_filter.cpp index 97b38e3d..7d66053b 100644 --- a/src/string_converter_filter.cpp +++ b/src/string_converter_filter.cpp @@ -1229,6 +1229,170 @@ JINJA2CPP_ALWAYS_INLINE InternalValue TitleCase(std::basic_string_view st return TitleCaseUnicode(str); } +// A cased character that str.lower() changes, or that str.upper() changes; characters with no +// case (digits, CJK) are neither. A titlecase character counts as uppercase. +inline bool ChangesUnder(CaseFn map, uint32_t cp) +{ + uint32_t mapped[3]; + return unicode::IsCased(cp) && (map(cp, mapped) != 1 || mapped[0] != cp); +} + +inline bool IsUpperChar(uint32_t cp) +{ + return ChangesUnder(unicode::ToLower, cp); +} + +inline bool IsLowerChar(uint32_t cp) +{ + return !IsUpperChar(cp) && ChangesUnder(unicode::ToUpper, cp); +} + +// The words camelize and underscorize build an identifier from, as [begin, end) indexes of +// chars: runs of \w characters other than _, split before an uppercase character that follows +// a lowercase one or a digit (fooBar, Vector3D), and before the last of a run of uppercase +// ones that a lowercase one follows (HTTPServer is HTTP and Server) +template +std::vector> IdentifierWords(const CaseChars& chars) +{ + const auto& cps = chars.cps; + auto inWord = [&cps](size_t idx) { return cps[idx] != '_' && unicode::IsWordChar(cps[idx]); }; + std::vector> words; + for (size_t idx = 0; idx < cps.size();) + { + if (!inWord(idx)) + { + ++idx; + continue; + } + auto begin = idx++; + for (; idx < cps.size() && inWord(idx); ++idx) + { + if (!IsUpperChar(cps[idx])) + { + continue; + } + const bool afterLower = IsLowerChar(cps[idx - 1]) || unicode::IsDecimal(cps[idx - 1]); + const bool acronymEnd = IsUpperChar(cps[idx - 1]) && idx + 1 < cps.size() && IsLowerChar(cps[idx + 1]); + if (afterLower || acronymEnd) + { + words.emplace_back(begin, idx); + begin = idx; + } + } + words.emplace_back(begin, idx); + } + return words; +} + +// camelize: the words of IdentifierWords, each with a titlecase first character and lowercase +// others, joined; with upperFirst false the first word is all lowercase ("http_server" and +// "HTTPServer" are HttpServer, or httpServer) +template +InternalValue Camelize(std::basic_string_view str, bool upperFirst) +{ + const CaseChars chars(str); + std::basic_string result; + bool first = true; + for (const auto& [begin, end] : IdentifierWords(chars)) + { + if (first && !upperFirst) + { + AppendLower(result, chars, begin, begin, end); + } + else + { + AppendMapped(result, chars, begin, unicode::ToTitle); + AppendLower(result, chars, begin, begin + 1, end); + } + first = false; + } + return InternalValue(result); +} + +// underscorize: the words of IdentifierWords in lowercase, joined with _ ("HTTPServer" and +// "http-server" are http_server) +template +InternalValue Underscorize(std::basic_string_view str) +{ + const CaseChars chars(str); + std::basic_string result; + for (const auto& [begin, end] : IdentifierWords(chars)) + { + if (!result.empty()) + { + result.push_back('_'); + } + AppendLower(result, chars, begin, begin, end); + } + return InternalValue(result); +} + +// escapecpp: the text as the contents of a C++ string literal. Backslash, double quote and +// control characters are escaped, the latter as \n-style escapes or three octal digits (an +// octal escape ends after three, where \x would swallow the hex digits that follow); other +// characters, non-ASCII ones too, are kept as they are. +template +std::basic_string EscapeCpp(std::basic_string_view str) +{ + std::basic_string result; + result.reserve(str.size()); + auto append = [&result](const char* text) { + for (; *text; ++text) + { + result.push_back(static_cast(*text)); + } + }; + for (auto ch : str) + { + const auto unit = CodeUnit(ch); + switch (unit) + { + case '\\': + append("\\\\"); + break; + case '"': + append("\\\""); + break; + case '\n': + append("\\n"); + break; + case '\r': + append("\\r"); + break; + case '\t': + append("\\t"); + break; + case '\a': + append("\\a"); + break; + case '\b': + append("\\b"); + break; + case '\f': + append("\\f"); + break; + case '\v': + append("\\v"); + break; + default: + if (unit < 0x20 || unit == 0x7F) + { + result.push_back('\\'); + for (int shift = 6; shift >= 0; shift -= 3) + { + result.push_back(static_cast('0' + ((unit >> shift) & 7))); + } + } + else + { + result.push_back(ch); + } + break; + } + } + return result; +} + // The code point a numeric character reference stands for, as html.unescape maps it; 0 for // the references it drops (html._invalid_codepoints) inline uint32_t CharRefCodePoint(uint64_t value) @@ -1539,6 +1703,12 @@ StringConverter::StringConverter(const FilterParams& params, StringConverter::Mo ParseParams(args, params); break; } + case CamelMode: + { + static const auto args = MakeArgumentsTable({ { "upper_first", false, true } }); + ParseParams(args, params); + break; + } default: ParseParams(NoArguments(), params); break; @@ -1945,6 +2115,15 @@ InternalValue StringConverter::Convert(const InternalValue& baseVal, RenderConte return InternalValue(ApplyCenter(baseVal, context)); case WordWrapMode: return InternalValue(ApplyWordWrap(baseVal, context)); + case CamelMode: + { + const bool upperFirst = ConvertToBool(this->GetArgumentValue("upper_first", context)); + return ConvertText(baseVal, [upperFirst](auto srcStr) { return Camelize(srcStr, upperFirst); }); + } + case UnderscoreMode: + return ConvertText(baseVal, [](auto srcStr) { return Underscorize(srcStr); }); + case EscapeCppMode: + return InternalValue(ApplyStringConverter(baseVal, [](auto srcStr) -> TargetString { return EscapeCpp(srcStr); })); default: return InternalValue(TargetString()); } @@ -2012,6 +2191,9 @@ InternalValue StringConverter::Filter(const InternalValue& baseVal, RenderContex case ReplaceMode: case WordCountMode: case StriptagsMode: + case CamelMode: + case UnderscoreMode: + case EscapeCppMode: // These filters convert any value with str() first if (!IsStringValue(baseVal)) { diff --git a/test/filters_test.cpp b/test/filters_test.cpp index c69794a4..914803a0 100644 --- a/test/filters_test.cpp +++ b/test/filters_test.cpp @@ -1,5 +1,6 @@ #include "test_tools.h" +#include #include #include #include @@ -649,6 +650,58 @@ INSTANTIATE_TEST_SUITE_P(Center, FilterGenericTest, ::testing::Values( InputOutputPair{" 'x' | center(width=0) | pprint", "'x'"}, InputOutputPair{" ' x' | center(width=5) | pprint", "' x '"} )); + +// C++-only filters (docs/tasks/0233): identifier case conversion, C++ string-literal escaping +// and block-style YAML +INSTANTIATE_TEST_SUITE_P(Camelize, FilterGenericTest, ::testing::Values( + InputOutputPair{"'device_type' | camelize", "DeviceType"}, + InputOutputPair{"'device_type' | camelize(upper_first=false)", "deviceType"}, + InputOutputPair{"'HTTPServer' | camelize", "HttpServer"}, + InputOutputPair{"'XMLHttpRequest' | camelize(false)", "xmlHttpRequest"}, + InputOutputPair{"'__FOO_BAR__' | camelize", "FooBar"}, + InputOutputPair{"'some-name with spaces.x' | camelize", "SomeNameWithSpacesX"}, + InputOutputPair{"'vector3d_type' | camelize", "Vector3dType"}, + InputOutputPair{"'' | camelize | pprint", "''"}, + InputOutputPair{"42 | camelize | pprint", "'42'"} + )); + +INSTANTIATE_TEST_SUITE_P(Underscorize, FilterGenericTest, ::testing::Values( + InputOutputPair{"'DeviceType' | underscorize", "device_type"}, + InputOutputPair{"'HTTPServer' | underscorize", "http_server"}, + InputOutputPair{"'getHTTPResponseCode' | underscorize", "get_http_response_code"}, + InputOutputPair{"'Vector3D' | underscorize", "vector3_d"}, + InputOutputPair{"'already_snake' | underscorize", "already_snake"}, + InputOutputPair{"' Some Name-here ' | underscorize", "some_name_here"}, + InputOutputPair{"'__init__' | underscorize", "init"}, + InputOutputPair{"'deviceType' | camelize | underscorize", "device_type"} + )); + +INSTANTIATE_TEST_SUITE_P(EscapeCpp, FilterGenericTest, ::testing::Values( + InputOutputPair{"'say \"hi\"' | escapecpp", "say \\\"hi\\\""}, + InputOutputPair{"'a\\\\b' | escapecpp", "a\\\\b"}, + InputOutputPair{"'tab\\tnl\\ncr\\r' | escapecpp", "tab\\tnl\\ncr\\r"}, + InputOutputPair{"\"it's ok\" | escapecpp", "it's ok"}, + InputOutputPair{"12 | escapecpp", "12"} + )); + +INSTANTIATE_TEST_SUITE_P(ToYaml, FilterGenericTest, ::testing::Values( + InputOutputPair{"{'a': 1, 'b': [1, 2], 'c': {'d': none, 'e': 'x y'}} | toyaml", + "a: 1\nb:\n - 1\n - 2\nc:\n d: null\n e: x y"}, + InputOutputPair{"[1, 'two', [3, 4], {'k': true}, [], {}] | toyaml", + "- 1\n- two\n- - 3\n - 4\n- k: true\n- []\n- {}"}, + InputOutputPair{"[{'a': 1, 'b': {'c': 2}}] | toyaml", "- a: 1\n b:\n c: 2"}, + InputOutputPair{"{'items': [{'id': 1}, {'id': 2}, {'n': 3}], 'e': []} | toyaml", "items:\n - id: 1\n - id: 2\n - \"n\": 3\ne: []"}, + InputOutputPair{"['yes', '', '1.5', ' x', 'x ', 'a: b', 'a#b', 'Null', '-x', 'a\\nb', 'q\"\\\\'] | toyaml", + "- \"yes\"\n- \"\"\n- \"1.5\"\n- \" x\"\n- \"x \"\n- \"a: b\"\n- \"a#b\"\n- \"Null\"\n- \"-x\"\n- \"a\\nb\"\n- \"q\\\"\\\\\""}, + InputOutputPair{"['plain text', 'path/to-file.txt', 'f(x) + 1', '_x'] | toyaml", + "- plain text\n- path/to-file.txt\n- f(x) + 1\n- _x"}, + InputOutputPair{"[1.5, 1e20, -2.0, false] | toyaml", "- 1.5\n- 1.0e+20\n- -2.0\n- false"}, + InputOutputPair{"{'a': 1} | dictsort | toyaml", "- - a\n - 1"}, + InputOutputPair{"'plain' | toyaml", "plain"}, + InputOutputPair{"none | toyaml", "null"}, + InputOutputPair{"{} | toyaml", "{}"}, + InputOutputPair{"[] | toyaml", "[]"} + )); // clang-format on struct XmlAttr : ::testing::Test { @@ -1032,6 +1085,23 @@ TEST(FilterErrors, VerifierLeftovers) EXPECT_EQ("0|0.0|5|5.0", RenderOrError("{{ fs|int }}|{{ us|float }}|{{ '\v5\t'|int }}|{{ '\f5\r'|float }}", { { "fs", fileSep }, { "us", unitSep } })); } +// docs/tasks/0233: the C++-only filters on non-ASCII text, control characters and values +// YAML has no form for; toxml was removed and is an unknown filter, as in Jinja2 +TEST(CppOnlyFilters, UnicodeControlsAndErrors) +{ + EXPECT_EQ("Stra\303\237eName|\303\244pfel_birnen|\303\204pfel", RenderOrError("{{ 'stra\303\237e_name'|camelize }}|{{ '\303\204pfelBirnen'|underscorize }}|{{ '\303\244pfel'|camelize }}", {})); + // Joined pieces: a hex escape would swallow the b and the 1 that follow it + const std::string controls = std::string("a\x01") + "b\x7f\x1b" + "1"; + EXPECT_EQ("a\\001b\\177\\0331|\303\251", RenderOrError("{{ s|escapecpp }}|{{ '\303\251'|escapecpp }}", { { "s", controls } })); + EXPECT_EQ("- \346\227\245\346\234\254\n- \"\\x01\\e\\0\"\n- \"\\u2028\"\n- .nan\n- -.inf", RenderOrError("{{ ['\346\227\245\346\234\254', s, '\342\200\250', 'nan'|float, '-inf'|float]|toyaml }}", { { "s", std::string("\x01\x1b\0", 3) } })); + EXPECT_NE(RenderOrError("{{ namespace(a=1)|toyaml }}", {}).find("not YAML serializable"), std::string::npos); + + Template tpl; + auto loaded = tpl.Load("{{ {'a': 1}|toxml }}"); + ASSERT_FALSE(loaded.has_value()); + EXPECT_EQ(ErrorCode::UnknownFilter, loaded.error().GetCode()); +} + TEST(FilterUnicode, PercentAEscapesNonAscii) { EXPECT_EQ("'\\xe9'", RenderOrError("{{ '%a'|format('\303\251') }}", {})); diff --git a/test/parity/cases/filters.py b/test/parity/cases/filters.py index 562eeb3d..83aa22d8 100644 --- a/test/parity/cases/filters.py +++ b/test/parity/cases/filters.py @@ -193,6 +193,8 @@ ("filter_kwargs_only", "{{ l|join(d=',') }}"), ("unknown_filter", "{{ x|nonexistent }}"), ("unknown_filter_unused_branch", "{% if false %}{{ x|nonexistent }}{% endif %}ok"), + # toxml was a C++-only filter until 2.0 (docs/tasks/0233) + ("unknown_filter_toxml", "{{ {'a': 1}|toxml }}"), # an unknown filter in code that never runs is still a load error in Jinja2 (docs/tasks/0193) ("unknown_filter_in_uncalled_macro", "{% macro m() %}{{ x|nonexistent }}{% endmacro %}ok"), ("unknown_filter_in_unused_default", "{% macro m(a=x|nonexistent) %}{{ a }}{% endmacro %}{{ m(1) }}"), diff --git a/test/parity/expected/filters.json b/test/parity/expected/filters.json index b6681a1e..a389e8a4 100644 --- a/test/parity/expected/filters.json +++ b/test/parity/expected/filters.json @@ -12146,6 +12146,76 @@ "templates": {}, "output": "ok" }, + { + "id": "filters.unknown_filter_toxml", + "template": "{{ {'a': 1}|toxml }}", + "context": { + "x": 3, + "neg": -3, + "y": 2.567, + "s": "Hello World", + "l": [ + 3, + 1, + 2 + ], + "dup": [ + 3, + 1, + 3, + 2, + 1 + ], + "words": [ + "b", + "A", + "c" + ], + "d": { + "b": 2, + "a": 1, + "C": 3 + }, + "e": [], + "n": null, + "html": "Tom & \"Jerry\"", + "long": "The quick brown fox jumps over the lazy dog", + "users": [ + { + "name": "bob", + "age": 30, + "city": "Oslo" + }, + { + "name": "alice", + "age": 25, + "city": "Rome" + }, + { + "name": "carl", + "age": 30, + "city": "Oslo" + }, + { + "name": "dan", + "age": 41 + } + ], + "lines": "a\nb\n\nc", + "nums_str": [ + "10", + "9", + "100" + ] + }, + "env": {}, + "templates": {}, + "error": { + "phase": "compile", + "type": "TemplateAssertionError", + "message": "No filter named 'toxml'." + } + }, { "id": "filters.unknown_filter_in_uncalled_macro", "template": "{% macro m() %}{{ x|nonexistent }}{% endmacro %}ok", From c33b101121d036bf85a0f423783e607bd2e283cc Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 18 Oct 2026 12:12:42 +0000 Subject: [PATCH 02/10] toyaml: decode malformed UTF-8 without an out-of-range shift, explicit long keys (0233) Found by the verifier: a lead byte followed by 32 or more continuation bytes shifted 0x7F past 31 bits in toyaml and tojson. Keys over 255 bytes are written as explicit '? key' entries, since YAML limits an implicit key to 1024 characters. The doc now says undefined values raise, as they do. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_017KNXpdwgCcK4gqUkYrVXzK --- docs/cpp-only-filters.md | 7 ++- docs/tasks/0233-cpp-only-filter-stubs.md | 2 +- src/serialize_filters.cpp | 64 +++++++++++++++--------- test/filters_test.cpp | 8 +++ 4 files changed, 54 insertions(+), 27 deletions(-) diff --git a/docs/cpp-only-filters.md b/docs/cpp-only-filters.md index fb976e27..28a626ec 100644 --- a/docs/cpp-only-filters.md +++ b/docs/cpp-only-filters.md @@ -52,7 +52,7 @@ const char* text = "{{ message|escapecpp }}"; A YAML document that YAML 1.1 and 1.2 loaders (PyYAML, yaml-cpp, ...) read back as the same data, with no trailing newline: -- `none` (and a plain undefined) is `null`, booleans `true`/`false`, integers in decimal, +- `none` is `null`, booleans `true`/`false`, integers in decimal, floats in Python's shortest form with a `.` always present (`1.0e+20`), `.nan`, `.inf` and `-.inf`. - A mapping is `key: value` lines in the mapping's order; a list is `- item` lines; a @@ -64,7 +64,10 @@ data, with no trailing newline: digits, spaces and `-.()+`, does not end with a space and is not a null or boolean word (`y`, `n`, `yes`, `no`, `on`, `off`, `true`, `false`, `null`, in any case). Any other string is double-quoted with YAML escapes (`\"`, `\\`, `\n`, `\t`, `\xHH`, `\uHHHH`, ...). -- `namespace()`, `cycler()`, `joiner()`, macros and strict undefined values raise an error. +- A key longer than 255 bytes is written as an explicit `? key` with its value on a `:` + line below, since YAML limits an implicit key to 1024 characters. +- Undefined values, `namespace()`, `cycler()`, `joiner()` and macros raise an error + ("Object of type Undefined is not YAML serializable"), as `tojson` does. ``` {{ {'name': 'demo', 'ports': [80, 443], 'tls': {'enabled': true}}|toyaml }} diff --git a/docs/tasks/0233-cpp-only-filter-stubs.md b/docs/tasks/0233-cpp-only-filter-stubs.md index 7ec93233..2333f860 100644 --- a/docs/tasks/0233-cpp-only-filter-stubs.md +++ b/docs/tasks/0233-cpp-only-filter-stubs.md @@ -31,4 +31,4 @@ migration notes (0077) say which. `escapecpp` and `toyaml` render the output documented in `docs/cpp-only-filters.md`, with rows in `test/filters_test.cpp`; `toxml` is removed and fails Load with `UnknownFilter` (corpus case `filters.unknown_filter_toxml`). Release note: `docs/api-2.0.md` 5.7, for -0077's `MIGRATION.md`. +0077's `MIGRATION.md`. PR #581. diff --git a/src/serialize_filters.cpp b/src/serialize_filters.cpp index 177c0745..17846259 100644 --- a/src/serialize_filters.cpp +++ b/src/serialize_filters.cpp @@ -192,6 +192,25 @@ Serialize::Serialize(const FilterParams& params, const Serialize::Mode mode) namespace { +// The code point of one character as SplitCodePoints splits UTF-8: the lead byte keeps 5, 4 or +// 3 bits by its own value, so malformed text (an ASCII lead or a long run of continuation +// bytes) gives some value rather than an out-of-range shift +uint32_t Utf8CharValue(std::string_view ch) +{ + const auto lead = static_cast(ch[0]); + if (lead < 0x80 || ch.size() == 1) + { + return lead; + } + uint32_t cp = lead & (lead >= 0xF0 ? 0x07U : lead >= 0xE0 ? 0x0FU + : 0x1FU); + for (size_t n = 1; n < ch.size(); ++n) + { + cp = (cp << 6) | (static_cast(ch[n]) & 0x3FU); + } + return cp; +} + // Python's json.dumps(value, sort_keys=True, indent=indent) with the default ensure_ascii, as // Jinja2's htmlsafe_json_dumps escapes it: <, >, & and ' become \u escapes. Outside strings // those characters can only come from the indent, which is escaped once up front. @@ -314,16 +333,7 @@ class PythonJsonWriter void WriteChar(std::string_view ch) { - // The lead byte keeps 7, 5, 4 or 3 bits for 1 to 4 byte sequences - uint32_t cp = static_cast(ch[0]); - if (ch.size() > 1) - { - cp &= 0x7FU >> ch.size(); - } - for (size_t n = 1; n < ch.size(); ++n) - { - cp = (cp << 6) | (static_cast(ch[n]) & 0x3F); - } + const auto cp = Utf8CharValue(ch); switch (cp) { case '"': @@ -479,6 +489,10 @@ class PythonJsonWriter class YamlWriter { public: + // Key bytes past which a key is written explicit: quoting expands a byte to at most four + // (\xHH), so 255 bytes and the quotes stay within the 1024 characters of an implicit key + static constexpr size_t MaxImplicitKey = 255; + explicit YamlWriter(RenderContext& context) : m_context(context) { @@ -511,16 +525,7 @@ class YamlWriter ++end; } const auto ch = str.substr(pos, end - pos); - uint32_t cp = static_cast(ch[0]); - if (ch.size() > 1) - { - cp &= 0x7FU >> ch.size(); - } - for (size_t n = 1; n < ch.size(); ++n) - { - cp = (cp << 6) | (static_cast(ch[n]) & 0x3F); - } - fn(cp, ch); + fn(Utf8CharValue(ch), ch); pos = end; } } @@ -533,10 +538,10 @@ class YamlWriter { return false; } - static const char* const words[] = { "y", "n", "yes", "no", "on", "off", "true", "false", "null" }; - for (const auto* word : words) + static const std::string_view words[] = { "y", "n", "yes", "no", "on", "off", "true", "false", "null" }; + for (const auto word : words) { - if (str.size() == std::strlen(word) && std::equal(str.begin(), str.end(), word, [](char l, char r) { return (l | 0x20) == r; })) + if (std::equal(str.begin(), str.end(), word.begin(), word.end(), [](char l, char r) { return (l | 0x20) == r; })) { return false; } @@ -793,7 +798,18 @@ class YamlWriter NewLine(indent); } isFirst = false; - WriteString(key); + if (key.size() > MaxImplicitKey) + { + // YAML limits an implicit key to 1024 characters; a longer one is "? key" with + // the value on a ": " line of its own + m_out += "? "; + WriteString(key); + NewLine(indent); + } + else + { + WriteString(key); + } m_out.push_back(':'); WriteNode(map.GetValueByName(key), indent + 2, Position::MappingValue); } diff --git a/test/filters_test.cpp b/test/filters_test.cpp index 914803a0..40f078d0 100644 --- a/test/filters_test.cpp +++ b/test/filters_test.cpp @@ -1095,6 +1095,14 @@ TEST(CppOnlyFilters, UnicodeControlsAndErrors) EXPECT_EQ("a\\001b\\177\\0331|\303\251", RenderOrError("{{ s|escapecpp }}|{{ '\303\251'|escapecpp }}", { { "s", controls } })); EXPECT_EQ("- \346\227\245\346\234\254\n- \"\\x01\\e\\0\"\n- \"\\u2028\"\n- .nan\n- -.inf", RenderOrError("{{ ['\346\227\245\346\234\254', s, '\342\200\250', 'nan'|float, '-inf'|float]|toyaml }}", { { "s", std::string("\x01\x1b\0", 3) } })); EXPECT_NE(RenderOrError("{{ namespace(a=1)|toyaml }}", {}).find("not YAML serializable"), std::string::npos); + EXPECT_NE(RenderOrError("{{ {'a': missing}|toyaml }}", {}).find("Object of type Undefined is not YAML serializable"), std::string::npos); + // Malformed UTF-8: a long run of continuation bytes was an out-of-range shift (UBSan) + const std::string tails = std::string("a") + std::string(40, '\x80'); + EXPECT_EQ(tails + "|\"a\"", RenderOrError("{{ s|toyaml }}|{{ s|tojson }}", { { "s", tails } })); + // Lazy sequences are iterated once; a key over 255 bytes is an explicit "? key" + EXPECT_EQ("- 0\n- 2|r:\n - 0\n - 1", RenderOrError("{{ range(0, 3, 2)|toyaml }}|{{ {'r': range(2)}|toyaml }}", {})); + const std::string longKey(300, 'k'); + EXPECT_EQ("? " + longKey + "\n: 1\n" + "short:\n - ? " + longKey + "\n :\n - 2", RenderOrError("{{ {k: 1, 'short': [{k: [2]}]}|toyaml }}", { { "k", longKey } })); Template tpl; auto loaded = tpl.Load("{{ {'a': 1}|toxml }}"); From 2964be5ea6244ee720a777271fedc1735349f514 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 18 Oct 2026 14:54:23 +0000 Subject: [PATCH 03/10] Add jinja2::BigInt as the last Value alternative (0269) Value can now hold an integer outside int64: jinja2::BigInt keeps its canonical decimal text (TryParse, Sign, ToInt64/ToUInt64, comparisons by value). The integral constructor, Reflect and the three JSON bindings store a BigInt for a value above INT64_MAX instead of throwing std::overflow_error (0172). sizeof(Value) is unchanged (40 B). Value gains IsBigInt, AsBigInt and the pointer form; ToString, ToChars and std::formatter print the digits. Until 0270 a BigInt that enters a template fails the render with std::overflow_error. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VLuNcuiZnzXVsxhZ3MzE9V --- CMakeLists.txt | 1 + docs/api-2.0.md | 8 +- include/jinja2cpp/big_int.h | 182 ++++++++++++++++++++ include/jinja2cpp/binding/boost_json.h | 4 +- include/jinja2cpp/binding/nlohmann_json.h | 4 +- include/jinja2cpp/binding/rapid_json.h | 2 +- include/jinja2cpp/reflected_value.h | 6 +- include/jinja2cpp/value.h | 30 +++- src/error_info.cpp | 2 + src/value.cpp | 7 + src/value_visitors.h | 12 ++ test/basic_tests.cpp | 60 ++++++- test/big_int_test.cpp | 134 ++++++++++++++ test/binding/boost_json_binding_test.cpp | 8 +- test/binding/nlohmann_json_binding_test.cpp | 8 +- test/binding/rapid_json_binding_test.cpp | 8 +- 16 files changed, 441 insertions(+), 35 deletions(-) create mode 100644 include/jinja2cpp/big_int.h create mode 100644 test/big_int_test.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index 7cb22ec9..620d6f60 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -369,6 +369,7 @@ if (JINJA2CPP_BUILD_TESTS) set(JINJA2CPP_TEST_SOURCES test/autoescape_test.cpp test/basic_tests.cpp + test/big_int_test.cpp test/containers_api_test.cpp test/errors_test.cpp test/expressions_test.cpp diff --git a/docs/api-2.0.md b/docs/api-2.0.md index 63671aa1..b8e1af10 100644 --- a/docs/api-2.0.md +++ b/docs/api-2.0.md @@ -136,6 +136,7 @@ the migration notes fix it), **fix** (a defect; behaviour changes, no rename), | `Value(T&&)` for unsigned, `long long`, `short`, `char`, `size_t` | *probe:* stores `bool` at C++14 (`Value(5u).data().index() == 1`), `int64_t` at C++17, and `Value(size_t(5))` does not compile at C++17 | constructors for every integral type storing `int64_t`, as `Reflect` already does; `Value(char)` deleted (ambiguous between a number and a one-letter string) | fix (task 0067) | | `Value(nullptr)` | *probe:* picks `Value(const char*)` and throws `logic_error` from `std::string(nullptr)` | `Value(std::nullptr_t)` makes `None` | fix (0067) | | `Value(ValuesList&&) noexcept`, `Value(ValuesMap&&) noexcept` | allocate, so `bad_alloc` becomes `std::terminate` | drop `noexcept` there | fix | +| `Value(uint64_t)` above `INT64_MAX` | cannot be held: wrapped to a negative number in 1.x, `std::overflow_error` since 0172 | `jinja2::BigInt` (`big_int.h`): an integer of any size as canonical decimal text, `TryParse(std::string_view, BigInt&) noexcept`, `Sign`, `ToInt64`/`ToUInt64`, comparisons by value; the last `Value` alternative (`sizeof(Value)` unchanged), with `IsBigInt()`, `AsBigInt()`, `bool AsBigInt(const BigInt*&)`; the integral constructor, `Reflect` and the JSON bindings store it for a value outside `int64_t`. `IsInt()` stays `int64_t` only. Until 0270 a `BigInt` that enters a template fails the render; arithmetic on it needs a backend (0271, plan docs/design/0178-big-integer-plan.md) | add (0269); an exhaustive `std::visit` over `GetData()` must handle the new alternative | | `IsEqual()` next to `operator==` | redundant | keep both; not worth a break | — | | `ValuesMap : std::unordered_map` | hash order | ordered map (task 0043) | break, done in 0043 | | free `AsString`/`AsWString` overloads for `std::string`, `std::wstring` and the views | encoding conversion; the `std::string` one returns a `const` value, defeating moves | `ToString`/`ToWString` returning `std::string`/`std::wstring` | alias | @@ -251,13 +252,13 @@ class Value public: using ValueData = variant, RecWrapper, - GenericList, GenericMap, RecWrapper>; + GenericList, GenericMap, RecWrapper, BigInt>; Value(); Value(std::nullptr_t) noexcept; // None template::value>* = nullptr> - Value(I val) noexcept(!detail::CanExceedInt64) // above INT64_MAX: std::overflow_error (0172) - : m_data(detail::ToStoredInt(val)) {} + Value(I val) noexcept(!detail::CanExceedInt64) // outside int64_t: a BigInt (0269) + : m_data(detail::ToStoredInt(val)) {} Value(char) = delete; // number or one-letter string? say which // ... string, double, list, map, callable constructors as today @@ -270,6 +271,7 @@ public: bool IsList() const noexcept; // ValuesList or GenericList bool IsMap() const noexcept; // ValuesMap or GenericMap bool IsCallable() const noexcept; + bool IsBigInt() const noexcept; // an integer outside int64_t const std::string& AsString() const; // throws bad_variant_access std::string& AsString(); diff --git a/include/jinja2cpp/big_int.h b/include/jinja2cpp/big_int.h new file mode 100644 index 00000000..b8d38178 --- /dev/null +++ b/include/jinja2cpp/big_int.h @@ -0,0 +1,182 @@ +#ifndef JINJA2CPP_BIG_INT_H +#define JINJA2CPP_BIG_INT_H + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace jinja2 +{ +namespace detail +{ +//! Integral types a \ref BigInt is made from: all except `bool` and the character types, as for \ref Value +template +inline constexpr bool IsBigIntSource = std::is_integral_v && !std::is_same_v && !std::is_same_v && !std::is_same_v && + !std::is_same_v && !std::is_same_v +#ifdef __cpp_char8_t + && !std::is_same_v +#endif + ; +} // namespace detail + +/*! + * \brief An integer of any size, as Python's `int` + * + * Holds the canonical decimal form: an optional `-` and digits without leading zeros (`"0"` for zero, never + * `"-0"`), so two equal numbers always have the same text. It does no arithmetic: it carries a number that does + * not fit `int64_t` between the host and a template. \ref Value stores one for an integer above `INT64_MAX` + * (a `uint64_t`, a JSON number), and a host may store one of any size. + * + * A `BigInt` may hold a number that fits `int64_t` (`BigInt(5)`); \ref Value keeps it as given. + */ +class BigInt +{ +public: + //! Zero + BigInt() = default; + + //! The value of any integral type except `bool` and the character types + template>* = nullptr> + explicit BigInt(I value) + { + using Unsigned = std::make_unsigned_t; + const bool negative = value < I(0); + // Negating in the unsigned type is exact for the most negative value too + Unsigned magnitude = negative ? static_cast(Unsigned(0) - static_cast(value)) : static_cast(value); + // 40 digits hold any 128-bit value + char buffer[48]; + char* end = buffer + sizeof(buffer); + char* begin = end; + do + { + *--begin = static_cast('0' + static_cast(magnitude % 10)); + magnitude /= 10; + } while (magnitude != 0); + if (negative) + { + *--begin = '-'; + } + m_decimal.assign(begin, end); + } + + /*! + * \brief Reads `[-]digits` into `out`, as Python's `int()` reads a decimal string with no spaces, sign `+` or + * underscores; returns whether it did + * + * Leading zeros are dropped (`"007"` is 7) and `"-0"` is 0. Anything else, the empty string included, returns + * `false` and leaves `out` unchanged, as does running out of memory for a long number. + */ + static bool TryParse(std::string_view decimal, BigInt& out) noexcept + { + const bool negative = !decimal.empty() && decimal.front() == '-'; + std::string_view digits = negative ? decimal.substr(1) : decimal; + if (digits.empty()) + { + return false; + } + for (const char c : digits) + { + if (c < '0' || c > '9') + { + return false; + } + } + const auto first = digits.find_first_not_of('0'); + if (first == std::string_view::npos) + { + out.m_decimal.assign(1, '0'); + return true; + } + digits.remove_prefix(first); + try + { + std::string text; + text.reserve(digits.size() + (negative ? 1 : 0)); + if (negative) + { + text.push_back('-'); + } + text.append(digits); + out.m_decimal = std::move(text); + return true; + } + catch (...) + { + return false; + } + } + + //! The canonical decimal form, as Python's `str()` prints the number + [[nodiscard]] const std::string& ToString() const noexcept { return m_decimal; } + + //! -1, 0 or 1 + [[nodiscard]] int Sign() const noexcept + { + if (m_decimal.front() == '-') + { + return -1; + } + return m_decimal == "0" ? 0 : 1; + } + + //! Stores the value in `out` and returns `true` when it fits `int64_t`; returns `false` otherwise + bool ToInt64(int64_t& out) const noexcept { return Load(out); } + + //! Stores the value in `out` and returns `true` when it fits `uint64_t`; returns `false` otherwise + bool ToUInt64(uint64_t& out) const noexcept { return Sign() >= 0 && Load(out); } + + //! Compares by value + friend bool operator==(const BigInt& lhs, const BigInt& rhs) noexcept { return lhs.m_decimal == rhs.m_decimal; } + friend bool operator!=(const BigInt& lhs, const BigInt& rhs) noexcept { return !(lhs == rhs); } + friend bool operator<(const BigInt& lhs, const BigInt& rhs) noexcept { return Compare(lhs, rhs) < 0; } + friend bool operator<=(const BigInt& lhs, const BigInt& rhs) noexcept { return Compare(lhs, rhs) <= 0; } + friend bool operator>(const BigInt& lhs, const BigInt& rhs) noexcept { return Compare(lhs, rhs) > 0; } + friend bool operator>=(const BigInt& lhs, const BigInt& rhs) noexcept { return Compare(lhs, rhs) >= 0; } + +private: + template + bool Load(T& out) const noexcept + { + T value{}; + const char* end = m_decimal.data() + m_decimal.size(); + const auto [ptr, ec] = std::from_chars(m_decimal.data(), end, value); + if (ec != std::errc() || ptr != end) + { + return false; + } + out = value; + return true; + } + + // Canonical forms compare by sign, then by length, then digit by digit + static int Compare(const BigInt& lhs, const BigInt& rhs) noexcept + { + const int lhsSign = lhs.Sign(); + const int rhsSign = rhs.Sign(); + if (lhsSign != rhsSign) + { + return lhsSign < rhsSign ? -1 : 1; + } + int magnitude = 0; + if (lhs.m_decimal.size() != rhs.m_decimal.size()) + { + magnitude = lhs.m_decimal.size() < rhs.m_decimal.size() ? -1 : 1; + } + else + { + const int cmp = lhs.m_decimal.compare(rhs.m_decimal); + magnitude = cmp < 0 ? -1 : (cmp > 0 ? 1 : 0); + } + return lhsSign < 0 ? -magnitude : magnitude; + } + + std::string m_decimal = "0"; +}; +} // namespace jinja2 + +#endif // JINJA2CPP_BIG_INT_H diff --git a/include/jinja2cpp/binding/boost_json.h b/include/jinja2cpp/binding/boost_json.h index 3a844927..61c2717b 100644 --- a/include/jinja2cpp/binding/boost_json.h +++ b/include/jinja2cpp/binding/boost_json.h @@ -184,7 +184,7 @@ struct Reflector result = val.get_int64(); break; case boost::json::kind::uint64: - result = detail::ToStoredInt(val.get_uint64()); + result = Value(val.get_uint64()); break; case boost::json::kind::double_: result = val.get_double(); @@ -223,7 +223,7 @@ struct Reflector result = val->get_int64(); break; case boost::json::kind::uint64: - result = detail::ToStoredInt(val->get_uint64()); + result = Value(val->get_uint64()); break; case boost::json::kind::double_: result = val->get_double(); diff --git a/include/jinja2cpp/binding/nlohmann_json.h b/include/jinja2cpp/binding/nlohmann_json.h index 6f0ebb80..6a8e7b78 100644 --- a/include/jinja2cpp/binding/nlohmann_json.h +++ b/include/jinja2cpp/binding/nlohmann_json.h @@ -174,7 +174,7 @@ struct NLohmannJsonReflector result = val.template get(); break; case nlohmann::detail::value_t::number_unsigned: - result = detail::ToStoredInt(val.template get()); + result = Value(val.template get()); break; case nlohmann::detail::value_t::number_float: result = val.template get(); @@ -210,7 +210,7 @@ struct NLohmannJsonReflector result = val->template get(); break; case nlohmann::detail::value_t::number_unsigned: - result = detail::ToStoredInt(val->template get()); + result = Value(val->template get()); break; case nlohmann::detail::value_t::number_float: result = val->template get(); diff --git a/include/jinja2cpp/binding/rapid_json.h b/include/jinja2cpp/binding/rapid_json.h index cc59321a..d2df1798 100644 --- a/include/jinja2cpp/binding/rapid_json.h +++ b/include/jinja2cpp/binding/rapid_json.h @@ -198,7 +198,7 @@ struct Reflector> } else if (val->IsUint64()) { - result = detail::ToStoredInt(val->GetUint64()); + result = Value(val->GetUint64()); } else if (val->IsInt() || val->IsUint()) { diff --git a/include/jinja2cpp/reflected_value.h b/include/jinja2cpp/reflected_value.h index 67d9d3cb..73bd3a67 100644 --- a/include/jinja2cpp/reflected_value.h +++ b/include/jinja2cpp/reflected_value.h @@ -661,12 +661,12 @@ struct Reflector>> }; // Every integral type except bool, including the character types, is reflected as int64_t; a value outside its -// range throws std::overflow_error, as Value's constructor does +// range becomes a BigInt, as Value's constructor stores it template struct Reflector && !std::is_same_v, bool>>> { - static auto Create(T val) { return Value(ToStoredInt(val)); } - static auto CreateFromPtr(const T* val) { return Value(ToStoredInt(*val)); } + static auto Create(T val) { return Value(ToStoredInt(val)); } + static auto CreateFromPtr(const T* val) { return Value(ToStoredInt(*val)); } }; } // namespace detail #endif diff --git a/include/jinja2cpp/value.h b/include/jinja2cpp/value.h index 9aeb49d8..637fa5a4 100644 --- a/include/jinja2cpp/value.h +++ b/include/jinja2cpp/value.h @@ -1,6 +1,7 @@ #ifndef JINJA2CPP_VALUE_H #define JINJA2CPP_VALUE_H +#include #include #include #include @@ -268,9 +269,10 @@ struct IsStoredAsInt : std::integral_constant && !st template inline constexpr bool CanExceedInt64 = std::numeric_limits::digits > std::numeric_limits::digits; -//! Converts an integral value to the stored `int64_t`; throws `std::overflow_error` for a value outside its range -template -constexpr int64_t ToStoredInt(T val) noexcept(!CanExceedInt64) +//! Converts an integral value to the alternative of `Data` (\ref Value::ValueData) that stores it: `int64_t`, or a +//! \ref BigInt for a value outside the `int64_t` range. Only types that can hold such values may throw (allocating) +template +Data ToStoredInt(T val) noexcept(!CanExceedInt64) { if constexpr (CanExceedInt64) { @@ -281,10 +283,10 @@ constexpr int64_t ToStoredInt(T val) noexcept(!CanExceedInt64) } if (!fits) { - throw std::overflow_error("jinja2cpp::Value: integer value is out of the int64_t range"); + return Data(BigInt(val)); } } - return static_cast(val); + return Data(static_cast(val)); } } // namespace detail @@ -301,6 +303,7 @@ constexpr int64_t ToStoredInt(T val) noexcept(!CanExceedInt64) * - String view value (std::string_view) * - Wide string view value (std::wstring_view) * - integer (int64_t) value + * - integer outside the int64_t range (\ref BigInt) * - floating point (double) value * - Simple list of other values (\ref ValuesList) * - Simple map of other values (\ref ValuesMap) @@ -328,7 +331,8 @@ class Value RecWrapper, GenericList, GenericMap, - RecWrapper>; + RecWrapper, + BigInt>; template struct AnyOf : public std::false_type @@ -415,14 +419,14 @@ class Value /*! * \brief Initializing constructor from any integral value except `bool` and characters: stores `int64_t` * - * A value outside the `int64_t` range (a `uint64_t` or `size_t` above `INT64_MAX`) throws `std::overflow_error` - * rather than wrapping; only types that can hold such values may throw. + * A value outside the `int64_t` range (a `uint64_t` or `size_t` above `INT64_MAX`) is stored as a \ref BigInt + * rather than wrapped; only types that can hold such values may throw (`std::bad_alloc`). * * @param val Integer value which should be used to initialize \ref Value instance */ template::value>* = nullptr> Value(I val) noexcept(!detail::CanExceedInt64) // NOLINT(google-explicit-constructor) - : m_data(detail::ToStoredInt(val)) + : m_data(detail::ToStoredInt(val)) { } /*! @@ -483,6 +487,8 @@ class Value [[nodiscard]] bool IsBool() const noexcept { return Holds(); } //! Test Value for holding an integer (`int64_t`) [[nodiscard]] bool IsInt() const noexcept { return Holds(); } + //! Test Value for holding an integer outside the `int64_t` range (\ref BigInt); \ref IsInt is false for it + [[nodiscard]] bool IsBigInt() const noexcept { return Holds(); } //! Test Value for holding a floating-point number (`double`) [[nodiscard]] bool IsDouble() const noexcept { return Holds(); } //! Test Value for holding `std::string` (a `std::string_view` does not count) @@ -517,6 +523,12 @@ class Value [[nodiscard]] int64_t AsInt() const { return As(); } //! Same as \ref AsInt(), without throwing bool AsInt(int64_t& out) const noexcept { return Load(out); } + //! The stored \ref BigInt; throws `std::bad_variant_access` for anything else, an `int64_t` included + [[nodiscard]] const BigInt& AsBigInt() const { return As(); } + //! A pointer to the stored \ref BigInt, without throwing or copying + bool AsBigInt(const BigInt*& out) const& noexcept { return LoadPtr(out); } + //! Deleted: the pointer would outlive the temporary it points into + bool AsBigInt(const BigInt*& out) && = delete; //! The stored `double`; throws `std::bad_variant_access` for anything else [[nodiscard]] double AsDouble() const { return As(); } //! Same as \ref AsDouble(), without throwing diff --git a/src/error_info.cpp b/src/error_info.cpp index 8da70a12..e826fea8 100644 --- a/src/error_info.cpp +++ b/src/error_info.cpp @@ -101,6 +101,8 @@ struct ValueRenderer void operator()(const jinja2::UserCallable& /*val*/) const {} + void operator()(const jinja2::BigInt& val) const { this->operator()(val.ToString()); } + template void operator()(const T& val) const { diff --git a/src/value.cpp b/src/value.cpp index 9594ad94..0782b3d8 100644 --- a/src/value.cpp +++ b/src/value.cpp @@ -21,6 +21,13 @@ namespace template void AppendText(const Value& val, std::basic_string& out) { + // A template cannot hold a big integer yet (docs/tasks/0270), but its text is all {{ value }} prints + if (const auto* big = val.GetIf()) + { + const auto& digits = big->ToString(); + out.append(digits.begin(), digits.end()); + return; + } const InternalValue internal = std::visit(visitors::InputValueConvertor(false, true), val.GetData()); visitors::RenderValue(internal, out, false); } diff --git a/src/value_visitors.h b/src/value_visitors.h index 5a66a30b..87815c9f 100644 --- a/src/value_visitors.h +++ b/src/value_visitors.h @@ -144,6 +144,12 @@ struct RecursiveUnwrapper { return arg.View(); } + // A host Value read in place may hold an integer outside int64, which templates cannot + // hold yet (docs/tasks/0270): it fails loudly, so no visitor meets one + [[noreturn]] static NoneValue UnwrapRecursive(const BigInt& arg, const ValueObject* /*owner*/) + { + throw std::overflow_error("integer " + arg.ToString() + " does not fit in 64 bits"); + } static ListRef UnwrapRecursive(const ListHandle& arg, const ValueObject* owner) noexcept { return ListRef(arg.accessor, owner, arg.isTuple); } static MapRef UnwrapRecursive(const MapHandle& arg, const ValueObject* owner) noexcept { return MapRef(arg.accessor, owner); } template @@ -633,6 +639,12 @@ struct InputValueConvertor return result_t(InternalValue(val)); } + // Integers outside int64 enter templates with docs/tasks/0270; until then they fail loudly + [[noreturn]] result_t operator()(const BigInt& val) const + { + throw std::overflow_error("integer " + val.ToString() + " does not fit in 64 bits"); + } + template result_t operator()(const T& val) const { diff --git a/test/basic_tests.cpp b/test/basic_tests.cpp index 3c46ba30..57111557 100644 --- a/test/basic_tests.cpp +++ b/test/basic_tests.cpp @@ -999,28 +999,63 @@ static_assert(std::is_nothrow_constructible_v); static_assert(std::is_nothrow_constructible_v); static_assert(!std::is_nothrow_constructible_v); -// An unsigned value above INT64_MAX throws instead of wrapping to a negative int64_t (task 0172) -TEST(BasicTests, ValueFromUnsignedAboveInt64MaxThrows) +// An unsigned value above INT64_MAX is stored as a BigInt instead of wrapping (task 0172, 0269) +TEST(BasicTests, ValueFromUnsignedAboveInt64MaxIsBigInt) { constexpr auto int64Max = std::numeric_limits::max(); constexpr auto above = static_cast(int64Max) + 1; + EXPECT_TRUE(Value(int64Max).IsInt()); EXPECT_EQ(int64Max, Value(static_cast(int64Max)).AsInt()); EXPECT_EQ(int64Max, Reflect(static_cast(int64Max)).AsInt()); - EXPECT_THROW(Value{ std::numeric_limits::max() }, std::overflow_error); - EXPECT_THROW(Value{ above }, std::overflow_error); - EXPECT_THROW(Value{ static_cast(-1) }, std::overflow_error); - EXPECT_THROW(Reflect(static_cast(-1)), std::overflow_error); - const uint64_t big = above; - EXPECT_THROW(Reflect(&big), std::overflow_error); - // A lazily reflected item fails the render loudly + const Value max{ std::numeric_limits::max() }; + EXPECT_TRUE(max.IsBigInt()); + EXPECT_FALSE(max.IsInt()); + EXPECT_EQ("18446744073709551615", max.AsBigInt().ToString()); + EXPECT_EQ("18446744073709551615", max.ToString()); + EXPECT_EQ(L"18446744073709551615", max.ToWString()); + EXPECT_EQ("9223372036854775808", Value{ above }.ToString()); + EXPECT_EQ("18446744073709551615", Value{ static_cast(-1) }.ToString()); + EXPECT_EQ("18446744073709551615", Reflect(static_cast(-1)).AsBigInt().ToString()); + const uint64_t big = above; + EXPECT_EQ("9223372036854775808", Reflect(&big).AsBigInt().ToString()); + + char buffer[32] = {}; + const auto [end, ec] = max.ToChars(std::begin(buffer), std::end(buffer)); + EXPECT_EQ(std::errc(), ec); + EXPECT_EQ("18446744073709551615", std::string_view(buffer, static_cast(end - buffer))); + + const BigInt* out = nullptr; + EXPECT_TRUE(max.AsBigInt(out)); + ASSERT_NE(nullptr, out); + EXPECT_EQ(BigInt(std::numeric_limits::max()), *out); + EXPECT_EQ(out, max.GetIf()); + const Value five(5); + EXPECT_FALSE(five.AsBigInt(out)); + EXPECT_EQ(nullptr, out); + EXPECT_THROW((void)Value(5).AsBigInt(), std::bad_variant_access); + int64_t asInt = 0; + EXPECT_FALSE(max.AsInt(asInt)); + + // Equal by value; an int64_t and a BigInt are different alternatives, as an int and a double are + BigInt parsed; + ASSERT_TRUE(BigInt::TryParse("18446744073709551615", parsed)); + EXPECT_EQ(max, Value(parsed)); + EXPECT_NE(max, Value(Value{ above })); + EXPECT_NE(Value(5), Value(BigInt(5))); + + // Until templates hold big integers (task 0270), one that enters a template fails the render loudly Template tpl; ASSERT_TRUE(tpl.Load("{{ items[0] }}")); const auto result = tpl.RenderAsString({ { "items", Reflect(std::vector{ above }) } }); ASSERT_FALSE(result.has_value()); EXPECT_EQ(ErrorCode::UnexpectedException, result.error().GetCode()); EXPECT_EQ("9223372036854775807", tpl.RenderAsString({ { "items", Reflect(std::vector{ static_cast(int64Max) }) } }).value()); + ASSERT_TRUE(tpl.Load("{{ n }}")); + const auto direct = tpl.RenderAsString({ { "n", max } }); + ASSERT_FALSE(direct.has_value()); + EXPECT_EQ(ErrorCode::UnexpectedException, direct.error().GetCode()); } TEST(BasicTests, ValueFromNullptrIsNone) @@ -1043,7 +1078,9 @@ TEST(ValueApiTest, PredicatesNameTheStoredType) EXPECT_TRUE(Value(Reflect(std::vector{ 1 })).IsList()); EXPECT_TRUE(Value(ValuesMap{ { "k", 1 } }).IsMap()); EXPECT_TRUE(Value(UserCallable()).IsCallable()); + EXPECT_TRUE(Value(BigInt(5)).IsBigInt()); + EXPECT_FALSE(Value(5).IsBigInt()); EXPECT_FALSE(Value(5).IsDouble()); EXPECT_FALSE(Value(true).IsInt()); EXPECT_FALSE(Value(std::string_view("s")).IsString()); @@ -1232,11 +1269,16 @@ TEST(ValueApiTest, VisitUnwrapsContainers) { return "int"; } + else if constexpr (std::is_same_v) + { + return "big " + item.ToString(); + } else { return "other"; } }; + EXPECT_EQ("big 18446744073709551615", Visit(kind, Value(std::numeric_limits::max()))); EXPECT_EQ("list of 2", Visit(kind, Value(ValuesList{ 1, 2 }))); EXPECT_EQ("map", Visit(kind, Value(ValuesMap{}))); EXPECT_EQ("callable", Visit(kind, Value(UserCallable()))); diff --git a/test/big_int_test.cpp b/test/big_int_test.cpp new file mode 100644 index 00000000..b466f74c --- /dev/null +++ b/test/big_int_test.cpp @@ -0,0 +1,134 @@ +#include "gtest/gtest.h" + +#include +#include + +#include +#include +#include +#include +#include +#include + +using namespace jinja2; + +// The public big integer (task 0269): canonical decimal text, no arithmetic + +// Adding the alternative leaves sizeof(Value) as it was: BigInt is no larger than the strings already in it +static_assert(sizeof(BigInt) == sizeof(std::string)); +static_assert(!std::is_constructible_v); +static_assert(!std::is_constructible_v); +static_assert(!std::is_constructible_v); +static_assert(!std::is_convertible_v); +static_assert(std::is_nothrow_move_constructible_v); + +TEST(BigIntTest, ConstructsFromIntegralTypes) +{ + EXPECT_EQ("0", BigInt().ToString()); + EXPECT_EQ("5", BigInt(5).ToString()); + EXPECT_EQ("-5", BigInt(-5).ToString()); + EXPECT_EQ("0", BigInt(0u).ToString()); + EXPECT_EQ("200", BigInt(static_cast(200)).ToString()); + EXPECT_EQ("-128", BigInt(static_cast(-128)).ToString()); + EXPECT_EQ("-32768", BigInt(std::numeric_limits::min()).ToString()); + EXPECT_EQ("9223372036854775807", BigInt(std::numeric_limits::max()).ToString()); + EXPECT_EQ("-9223372036854775808", BigInt(std::numeric_limits::min()).ToString()); + EXPECT_EQ("18446744073709551615", BigInt(std::numeric_limits::max()).ToString()); +} + +TEST(BigIntTest, TryParseAcceptsDecimalOnly) +{ + struct Row + { + std::string_view input; + const char* expected; // nullptr: rejected + }; + const Row rows[] = { + { "0", "0" }, + { "7", "7" }, + { "007", "7" }, + { "-0", "0" }, + { "-000", "0" }, + { "-007", "-7" }, + { "18446744073709551616", "18446744073709551616" }, + { "-123456789012345678901234567890", "-123456789012345678901234567890" }, + { "", nullptr }, + { "-", nullptr }, + { "+1", nullptr }, + { "1e3", nullptr }, + { " 1", nullptr }, + { "1 ", nullptr }, + { "1_000", nullptr }, + { "--1", nullptr }, + { "0x10", nullptr }, + { "1.0", nullptr }, + }; + for (const auto& row : rows) + { + BigInt out(42); + const bool parsed = BigInt::TryParse(row.input, out); + if (row.expected) + { + EXPECT_TRUE(parsed) << row.input; + EXPECT_EQ(row.expected, out.ToString()) << row.input; + } + else + { + EXPECT_FALSE(parsed) << row.input; + EXPECT_EQ("42", out.ToString()) << row.input << ": left unchanged"; + } + } +} + +TEST(BigIntTest, SignAndConversions) +{ + EXPECT_EQ(0, BigInt().Sign()); + EXPECT_EQ(1, BigInt(3).Sign()); + EXPECT_EQ(-1, BigInt(-3).Sign()); + + int64_t i = 0; + uint64_t u = 0; + EXPECT_TRUE(BigInt(std::numeric_limits::min()).ToInt64(i)); + EXPECT_EQ(std::numeric_limits::min(), i); + EXPECT_FALSE(BigInt(std::numeric_limits::max()).ToInt64(i)); + EXPECT_EQ(std::numeric_limits::min(), i); + EXPECT_TRUE(BigInt(std::numeric_limits::max()).ToUInt64(u)); + EXPECT_EQ(std::numeric_limits::max(), u); + EXPECT_FALSE(BigInt(-1).ToUInt64(u)); + BigInt huge; + ASSERT_TRUE(BigInt::TryParse("18446744073709551616", huge)); + EXPECT_FALSE(huge.ToUInt64(u)); + EXPECT_FALSE(huge.ToInt64(i)); + EXPECT_TRUE(BigInt().ToUInt64(u)); + EXPECT_EQ(0u, u); +} + +TEST(BigIntTest, ComparesByValue) +{ + // Ascending; every pair compares as its position does + std::vector sorted; + for (const char* text : { "-100000000000000000000", "-99999999999999999999", "-10", "-9", "0", "9", "10", "99999999999999999999", + "100000000000000000000" }) + { + BigInt value; + ASSERT_TRUE(BigInt::TryParse(text, value)) << text; + sorted.push_back(value); + } + for (size_t a = 0; a < sorted.size(); ++a) + { + for (size_t b = 0; b < sorted.size(); ++b) + { + const auto& lhs = sorted[a]; + const auto& rhs = sorted[b]; + EXPECT_EQ(a == b, lhs == rhs) << lhs.ToString() << " " << rhs.ToString(); + EXPECT_EQ(a != b, lhs != rhs) << lhs.ToString() << " " << rhs.ToString(); + EXPECT_EQ(a < b, lhs < rhs) << lhs.ToString() << " " << rhs.ToString(); + EXPECT_EQ(a <= b, lhs <= rhs) << lhs.ToString() << " " << rhs.ToString(); + EXPECT_EQ(a > b, lhs > rhs) << lhs.ToString() << " " << rhs.ToString(); + EXPECT_EQ(a >= b, lhs >= rhs) << lhs.ToString() << " " << rhs.ToString(); + } + } + BigInt padded; + ASSERT_TRUE(BigInt::TryParse("-0009", padded)); + EXPECT_EQ(BigInt(-9), padded); +} diff --git a/test/binding/boost_json_binding_test.cpp b/test/binding/boost_json_binding_test.cpp index 1f6086fd..abf3e9c9 100644 --- a/test/binding/boost_json_binding_test.cpp +++ b/test/binding/boost_json_binding_test.cpp @@ -221,12 +221,16 @@ TEST(BoostJsonAccessorTest, FindAndContains) EXPECT_TRUE(map.GetAccessor()->GetValueByName("absent").IsNone()); } -// A JSON unsigned number above INT64_MAX fails the render instead of wrapping (task 0172) -TEST(BoostJsonUnsignedTest, UnsignedAboveInt64MaxFails) +// A JSON unsigned number above INT64_MAX is a BigInt instead of wrapping (task 0172, 0269); until templates +// hold big integers (task 0270) the render that reads it fails +TEST(BoostJsonUnsignedTest, UnsignedAboveInt64MaxIsBigInt) { jinja2::Template tpl; ASSERT_TRUE(tpl.Load("{{ json.max }}|{{ json.big }}")); jinja2::ValuesMap params{ { "json", jinja2::Reflect(boost::json::parse(R"({"max": 9223372036854775807, "big": 18446744073709551615})")) } }; + const auto big = params["json"].ToGenericMap().GetValueByName("big"); + ASSERT_TRUE(big.IsBigInt()); + EXPECT_EQ("18446744073709551615", big.AsBigInt().ToString()); const auto result = tpl.RenderAsString(params); ASSERT_FALSE(result.has_value()); EXPECT_EQ(jinja2::ErrorCode::UnexpectedException, result.error().GetCode()); diff --git a/test/binding/nlohmann_json_binding_test.cpp b/test/binding/nlohmann_json_binding_test.cpp index f05bec52..92121e51 100644 --- a/test/binding/nlohmann_json_binding_test.cpp +++ b/test/binding/nlohmann_json_binding_test.cpp @@ -215,12 +215,16 @@ MULTISTR_TEST(NlohmannJsonTest, OrderedJsonKeepsDocumentOrder, R"({{ sorted }}|{ params["json"] = jinja2::Reflect(nlohmann::ordered_json::parse(doc)); } -// A JSON unsigned number above INT64_MAX fails the render instead of wrapping (task 0172) -TEST(NlohmannJsonUnsignedTest, UnsignedAboveInt64MaxFails) +// A JSON unsigned number above INT64_MAX is a BigInt instead of wrapping (task 0172, 0269); until templates +// hold big integers (task 0270) the render that reads it fails +TEST(NlohmannJsonUnsignedTest, UnsignedAboveInt64MaxIsBigInt) { jinja2::Template tpl; ASSERT_TRUE(tpl.Load("{{ json.max }}|{{ json.big }}")); jinja2::ValuesMap params{ { "json", jinja2::Reflect(nlohmann::json::parse(R"({"max": 9223372036854775807, "big": 18446744073709551615})")) } }; + const auto big = params["json"].ToGenericMap().GetValueByName("big"); + ASSERT_TRUE(big.IsBigInt()); + EXPECT_EQ("18446744073709551615", big.AsBigInt().ToString()); const auto result = tpl.RenderAsString(params); ASSERT_FALSE(result.has_value()); EXPECT_EQ(jinja2::ErrorCode::UnexpectedException, result.error().GetCode()); diff --git a/test/binding/rapid_json_binding_test.cpp b/test/binding/rapid_json_binding_test.cpp index fb745a47..d82ae125 100644 --- a/test/binding/rapid_json_binding_test.cpp +++ b/test/binding/rapid_json_binding_test.cpp @@ -141,14 +141,18 @@ MULTISTR_TEST(RapidJsonTest, NullMemberIsNone, R"({{ json.x is defined }}{{ json params["json"] = jinja2::Reflect(doc); } -// A JSON unsigned number above INT64_MAX fails the render instead of wrapping (task 0172) -TEST(RapidJsonUnsignedTest, UnsignedAboveInt64MaxFails) +// A JSON unsigned number above INT64_MAX is a BigInt instead of wrapping (task 0172, 0269); until templates +// hold big integers (task 0270) the render that reads it fails +TEST(RapidJsonUnsignedTest, UnsignedAboveInt64MaxIsBigInt) { rapidjson::Document doc; doc.Parse(R"({"max": 9223372036854775807, "big": 18446744073709551615})"); jinja2::Template tpl; ASSERT_TRUE(tpl.Load("{{ json.max }}|{{ json.big }}")); jinja2::ValuesMap params{ { "json", jinja2::Reflect(doc) } }; + const auto big = params["json"].ToGenericMap().GetValueByName("big"); + ASSERT_TRUE(big.IsBigInt()); + EXPECT_EQ("18446744073709551615", big.AsBigInt().ToString()); const auto result = tpl.RenderAsString(params); ASSERT_FALSE(result.has_value()); EXPECT_EQ(jinja2::ErrorCode::UnexpectedException, result.error().GetCode()); From 8e393fbb55d696da4c3cf44d1deedff2cfbab9f3 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 18 Oct 2026 14:55:49 +0000 Subject: [PATCH 04/10] Format big_int.h; take the digit buffer's end with std::end --- include/jinja2cpp/big_int.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/include/jinja2cpp/big_int.h b/include/jinja2cpp/big_int.h index b8d38178..bc4f7236 100644 --- a/include/jinja2cpp/big_int.h +++ b/include/jinja2cpp/big_int.h @@ -4,6 +4,7 @@ #include #include #include +#include #include #include #include @@ -16,8 +17,7 @@ namespace detail { //! Integral types a \ref BigInt is made from: all except `bool` and the character types, as for \ref Value template -inline constexpr bool IsBigIntSource = std::is_integral_v && !std::is_same_v && !std::is_same_v && !std::is_same_v && - !std::is_same_v && !std::is_same_v +inline constexpr bool IsBigIntSource = std::is_integral_v && !std::is_same_v && !std::is_same_v && !std::is_same_v && !std::is_same_v && !std::is_same_v #ifdef __cpp_char8_t && !std::is_same_v #endif @@ -50,7 +50,7 @@ class BigInt Unsigned magnitude = negative ? static_cast(Unsigned(0) - static_cast(value)) : static_cast(value); // 40 digits hold any 128-bit value char buffer[48]; - char* end = buffer + sizeof(buffer); + char* end = std::end(buffer); char* begin = end; do { From 8842e89579da41c55fb6db761e05307b1995d82a Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 18 Oct 2026 15:20:15 +0000 Subject: [PATCH 05/10] Include big_int.h where BigInt is used --- include/jinja2cpp/big_int.h | 1 - src/error_info.cpp | 1 + src/value.cpp | 1 + src/value_visitors.h | 1 + test/big_int_test.cpp | 2 +- 5 files changed, 4 insertions(+), 2 deletions(-) diff --git a/include/jinja2cpp/big_int.h b/include/jinja2cpp/big_int.h index bc4f7236..4a802de8 100644 --- a/include/jinja2cpp/big_int.h +++ b/include/jinja2cpp/big_int.h @@ -2,7 +2,6 @@ #define JINJA2CPP_BIG_INT_H #include -#include #include #include #include diff --git a/src/error_info.cpp b/src/error_info.cpp index e826fea8..55de9ba1 100644 --- a/src/error_info.cpp +++ b/src/error_info.cpp @@ -2,6 +2,7 @@ #include "helpers.h" +#include #include #include #include diff --git a/src/value.cpp b/src/value.cpp index 0782b3d8..d62bdefa 100644 --- a/src/value.cpp +++ b/src/value.cpp @@ -1,6 +1,7 @@ #include "internal_value.h" #include "value_visitors.h" +#include #include #include diff --git a/src/value_visitors.h b/src/value_visitors.h index 87815c9f..4b8b8dbc 100644 --- a/src/value_visitors.h +++ b/src/value_visitors.h @@ -9,6 +9,7 @@ #include "undefined.h" #include "unicode_printable.h" +#include #include #include #include diff --git a/test/big_int_test.cpp b/test/big_int_test.cpp index b466f74c..a39a8d7f 100644 --- a/test/big_int_test.cpp +++ b/test/big_int_test.cpp @@ -1,8 +1,8 @@ #include "gtest/gtest.h" #include -#include +#include #include #include #include From 354a7a030e61a86878e6b0848e85a03618691a90 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 18 Oct 2026 15:37:56 +0000 Subject: [PATCH 06/10] BigInt: take digits in a 64-bit or wider type; pin nested printing until 0270 Addresses the verifier's MSVC risk (C4244/C4296 on narrow integral types under /W4 /WX) and documents that ToString prints a BigInt only at top level until templates hold big integers. --- docs/api-2.0.md | 2 +- include/jinja2cpp/big_int.h | 17 +++++++++++++---- test/basic_tests.cpp | 3 +++ 3 files changed, 17 insertions(+), 5 deletions(-) diff --git a/docs/api-2.0.md b/docs/api-2.0.md index b8e1af10..92d99588 100644 --- a/docs/api-2.0.md +++ b/docs/api-2.0.md @@ -136,7 +136,7 @@ the migration notes fix it), **fix** (a defect; behaviour changes, no rename), | `Value(T&&)` for unsigned, `long long`, `short`, `char`, `size_t` | *probe:* stores `bool` at C++14 (`Value(5u).data().index() == 1`), `int64_t` at C++17, and `Value(size_t(5))` does not compile at C++17 | constructors for every integral type storing `int64_t`, as `Reflect` already does; `Value(char)` deleted (ambiguous between a number and a one-letter string) | fix (task 0067) | | `Value(nullptr)` | *probe:* picks `Value(const char*)` and throws `logic_error` from `std::string(nullptr)` | `Value(std::nullptr_t)` makes `None` | fix (0067) | | `Value(ValuesList&&) noexcept`, `Value(ValuesMap&&) noexcept` | allocate, so `bad_alloc` becomes `std::terminate` | drop `noexcept` there | fix | -| `Value(uint64_t)` above `INT64_MAX` | cannot be held: wrapped to a negative number in 1.x, `std::overflow_error` since 0172 | `jinja2::BigInt` (`big_int.h`): an integer of any size as canonical decimal text, `TryParse(std::string_view, BigInt&) noexcept`, `Sign`, `ToInt64`/`ToUInt64`, comparisons by value; the last `Value` alternative (`sizeof(Value)` unchanged), with `IsBigInt()`, `AsBigInt()`, `bool AsBigInt(const BigInt*&)`; the integral constructor, `Reflect` and the JSON bindings store it for a value outside `int64_t`. `IsInt()` stays `int64_t` only. Until 0270 a `BigInt` that enters a template fails the render; arithmetic on it needs a backend (0271, plan docs/design/0178-big-integer-plan.md) | add (0269); an exhaustive `std::visit` over `GetData()` must handle the new alternative | +| `Value(uint64_t)` above `INT64_MAX` | cannot be held: wrapped to a negative number in 1.x, `std::overflow_error` since 0172 | `jinja2::BigInt` (`big_int.h`): an integer of any size as canonical decimal text, `TryParse(std::string_view, BigInt&) noexcept`, `Sign`, `ToInt64`/`ToUInt64`, comparisons by value; the last `Value` alternative (`sizeof(Value)` unchanged), with `IsBigInt()`, `AsBigInt()`, `bool AsBigInt(const BigInt*&)`; the integral constructor, `Reflect` and the JSON bindings store it for a value outside `int64_t`. `IsInt()` stays `int64_t` only. Until 0270 a `BigInt` that enters a template fails the render (a top-level render parameter fails it even unread), and `ToString()` prints one only at top level, not inside a list or map; arithmetic on it needs a backend (0271, plan docs/design/0178-big-integer-plan.md) | add (0269); an exhaustive `std::visit` over `GetData()` must handle the new alternative | | `IsEqual()` next to `operator==` | redundant | keep both; not worth a break | — | | `ValuesMap : std::unordered_map` | hash order | ordered map (task 0043) | break, done in 0043 | | free `AsString`/`AsWString` overloads for `std::string`, `std::wstring` and the views | encoding conversion; the `std::string` one returns a `const` value, defeating moves | `ToString`/`ToWString` returning `std::string`/`std::wstring` | alias | diff --git a/include/jinja2cpp/big_int.h b/include/jinja2cpp/big_int.h index 4a802de8..e68f4816 100644 --- a/include/jinja2cpp/big_int.h +++ b/include/jinja2cpp/big_int.h @@ -43,10 +43,19 @@ class BigInt template>* = nullptr> explicit BigInt(I value) { - using Unsigned = std::make_unsigned_t; - const bool negative = value < I(0); - // Negating in the unsigned type is exact for the most negative value too - Unsigned magnitude = negative ? static_cast(Unsigned(0) - static_cast(value)) : static_cast(value); + // Digits are taken in a type no narrower than 64 bits, so that no step narrows through int + using Unsigned = std::conditional_t<(sizeof(I) > sizeof(uint64_t)), std::make_unsigned_t, uint64_t>; + bool negative = false; + Unsigned magnitude = static_cast(value); + if constexpr (std::is_signed_v) + { + negative = value < 0; + // Negating in the unsigned type is exact for the most negative value too + if (negative) + { + magnitude = Unsigned(0) - magnitude; + } + } // 40 digits hold any 128-bit value char buffer[48]; char* end = std::end(buffer); diff --git a/test/basic_tests.cpp b/test/basic_tests.cpp index 57111557..1c0cfe84 100644 --- a/test/basic_tests.cpp +++ b/test/basic_tests.cpp @@ -1021,6 +1021,9 @@ TEST(BasicTests, ValueFromUnsignedAboveInt64MaxIsBigInt) const uint64_t big = above; EXPECT_EQ("9223372036854775808", Reflect(&big).AsBigInt().ToString()); + // Inside a list or map it goes through the template renderer, which holds big integers only with task 0270 + EXPECT_THROW((void)Value(ValuesList{ max }).ToString(), std::overflow_error); + char buffer[32] = {}; const auto [end, ec] = max.ToChars(std::begin(buffer), std::end(buffer)); EXPECT_EQ(std::errc(), ec); From b2fb5a02f586bb14225e2c62db7b5f5427c051c2 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 18 Oct 2026 15:48:16 +0000 Subject: [PATCH 07/10] BigInt: use auto for the cast (modernize-use-auto) --- include/jinja2cpp/big_int.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/jinja2cpp/big_int.h b/include/jinja2cpp/big_int.h index e68f4816..4156779a 100644 --- a/include/jinja2cpp/big_int.h +++ b/include/jinja2cpp/big_int.h @@ -46,7 +46,7 @@ class BigInt // Digits are taken in a type no narrower than 64 bits, so that no step narrows through int using Unsigned = std::conditional_t<(sizeof(I) > sizeof(uint64_t)), std::make_unsigned_t, uint64_t>; bool negative = false; - Unsigned magnitude = static_cast(value); + auto magnitude = static_cast(value); if constexpr (std::is_signed_v) { negative = value < 0; From 65cec3e6961621260a9f4be1c29e30cf9c150011 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 18 Oct 2026 18:28:40 +0000 Subject: [PATCH 08/10] BigInt: a moved-from BigInt is zero (Codex P2) A defaulted move could leave the string empty, and Sign() then read front() of an empty string. The move operations now leave the source "0", so every BigInt holds a canonical number. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01VLuNcuiZnzXVsxhZ3MzE9V --- include/jinja2cpp/big_int.h | 19 +++++++++++++++++++ test/big_int_test.cpp | 27 +++++++++++++++++++++++++++ 2 files changed, 46 insertions(+) diff --git a/include/jinja2cpp/big_int.h b/include/jinja2cpp/big_int.h index 4156779a..8b1a42f8 100644 --- a/include/jinja2cpp/big_int.h +++ b/include/jinja2cpp/big_int.h @@ -38,6 +38,25 @@ class BigInt public: //! Zero BigInt() = default; + BigInt(const BigInt&) = default; + BigInt& operator=(const BigInt&) = default; + //! Leaves `other` zero, so that a moved-from BigInt still holds a canonical number + BigInt(BigInt&& other) noexcept + : m_decimal(std::move(other.m_decimal)) + { + other.m_decimal.assign(1, '0'); + } + //! Leaves `other` zero, as the move constructor does + BigInt& operator=(BigInt&& other) noexcept + { + if (this != &other) + { + m_decimal.swap(other.m_decimal); + other.m_decimal.assign(1, '0'); + } + return *this; + } + ~BigInt() = default; //! The value of any integral type except `bool` and the character types template>* = nullptr> diff --git a/test/big_int_test.cpp b/test/big_int_test.cpp index a39a8d7f..17e9b322 100644 --- a/test/big_int_test.cpp +++ b/test/big_int_test.cpp @@ -1,6 +1,7 @@ #include "gtest/gtest.h" #include +#include #include #include @@ -8,6 +9,7 @@ #include #include #include +#include #include using namespace jinja2; @@ -132,3 +134,28 @@ TEST(BigIntTest, ComparesByValue) ASSERT_TRUE(BigInt::TryParse("-0009", padded)); EXPECT_EQ(BigInt(-9), padded); } + +// A moved-from BigInt is zero, never an empty text that Sign() or a comparison would read past +TEST(BigIntTest, MovedFromIsZero) +{ + BigInt a(-12); + const BigInt b(std::move(a)); + EXPECT_EQ("-12", b.ToString()); + EXPECT_EQ("0", a.ToString()); // NOLINT(bugprone-use-after-move,clang-analyzer-cplusplus.Move) + EXPECT_EQ(0, a.Sign()); // NOLINT(bugprone-use-after-move,clang-analyzer-cplusplus.Move) + EXPECT_TRUE(a < b || b < a); // NOLINT(bugprone-use-after-move,clang-analyzer-cplusplus.Move) + + BigInt c(7); + c = std::move(a); // NOLINT(bugprone-use-after-move,clang-analyzer-cplusplus.Move) + EXPECT_EQ("0", c.ToString()); + EXPECT_EQ("0", a.ToString()); // NOLINT(bugprone-use-after-move,clang-analyzer-cplusplus.Move) + + // Moving a Value moves the BigInt it holds + Value held(std::numeric_limits::max()); + const Value moved(std::move(held)); + EXPECT_EQ("18446744073709551615", moved.ToString()); + if (const auto* rest = held.GetIf()) // NOLINT(bugprone-use-after-move,clang-analyzer-cplusplus.Move) + { + EXPECT_EQ(0, rest->Sign()); + } +} From 267fb3933aea319f0d76f8132bb34f93ded5b22b Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 18 Oct 2026 22:07:50 +0000 Subject: [PATCH 09/10] toyaml, tojson: stop an endless generator at the size cap (0233) Found by Codex: a host generator with no end grew the serialized text without bound. Both writers now check the text against MaxSequenceSize per item. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_017KNXpdwgCcK4gqUkYrVXzK --- src/serialize_filters.cpp | 5 +++++ test/filters_test.cpp | 8 ++++++++ 2 files changed, 13 insertions(+) diff --git a/src/serialize_filters.cpp b/src/serialize_filters.cpp index 17846259..16f26ecd 100644 --- a/src/serialize_filters.cpp +++ b/src/serialize_filters.cpp @@ -385,6 +385,8 @@ class PythonJsonWriter bool isFirst = true; for (auto& item : items) { + // A generator with no end must not grow the text past the cap + CheckSequenceSize(m_out.View().size(), "a JSON document"); if (!isFirst) { m_out += m_indent ? "," : ", "; @@ -661,8 +663,11 @@ class YamlWriter m_out += view; } + // Every item but a collection's first starts a line: the document stays within the cap + // a generator with no end, or quadratic indentation, would pass void NewLine(size_t indent) { + CheckSequenceSize(m_out.View().size(), "a YAML document"); m_out.push_back('\n'); for (size_t n = 0; n != indent; ++n) { diff --git a/test/filters_test.cpp b/test/filters_test.cpp index 40f078d0..e5399913 100644 --- a/test/filters_test.cpp +++ b/test/filters_test.cpp @@ -1,13 +1,16 @@ #include "test_tools.h" #include +#include #include #include #include #include #include +#include #include +#include #include #include #include @@ -1101,6 +1104,11 @@ TEST(CppOnlyFilters, UnicodeControlsAndErrors) EXPECT_EQ(tails + "|\"a\"", RenderOrError("{{ s|toyaml }}|{{ s|tojson }}", { { "s", tails } })); // Lazy sequences are iterated once; a key over 255 bytes is an explicit "? key" EXPECT_EQ("- 0\n- 2|r:\n - 0\n - 1", RenderOrError("{{ range(0, 3, 2)|toyaml }}|{{ {'r': range(2)}|toyaml }}", {})); + // A generator with no end stops at the size cap, in toyaml and tojson alike + const std::string chunk(size_t{ 1 } << 20, 'x'); + auto endless = [&chunk] { return MakeGenericList([&chunk]() -> std::optional { return Value(chunk); }); }; + EXPECT_NE(RenderOrError("{{ g|toyaml }}", { { "g", endless() } }).find("a YAML document of length"), std::string::npos); + EXPECT_NE(RenderOrError("{{ g|tojson }}", { { "g", endless() } }).find("a JSON document of length"), std::string::npos); const std::string longKey(300, 'k'); EXPECT_EQ("? " + longKey + "\n: 1\n" + "short:\n - ? " + longKey + "\n :\n - 2", RenderOrError("{{ {k: 1, 'short': [{k: [2]}]}|toyaml }}", { { "k", longKey } })); From b43017db8ded47cebeb1845507173e28d5e0c3e1 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 19 Oct 2026 00:40:04 +0000 Subject: [PATCH 10/10] Merge train 26: public BigInt value and C++-only filters Regenerate the task index and parity table. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01DTQTriEsFrrBgcYNKDUUF4 --- docs/parity.md | 6 +++--- docs/tasks/0269-public-big-int-value.md | 4 +++- docs/tasks/README.md | 4 ++-- 3 files changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/parity.md b/docs/parity.md index 2aa8e2b1..7e712d56 100644 --- a/docs/parity.md +++ b/docs/parity.md @@ -4,7 +4,7 @@ How far Jinja2C++ is from Python [Jinja2](https://jinja.palletsprojects.com/) 3. area, and which task in `docs/tasks/` closes each gap. Every statement marked with a case id is backed by the differential corpus in -`test/parity/` (1938 templates rendered by both engines, see +`test/parity/` (1939 templates rendered by both engines, see [test/parity/README.md](../test/parity/README.md)); `ctest -R parity` re-checks all of them. Statements in the last section (API level) are read from the headers and are not corpus-checked yet. @@ -21,7 +21,7 @@ Snapshot of `python3 test/parity/generate.py --report` (Jinja2 3.1.6, Oct 2026): | autoescape | 124 | 122 | 1 | 1 | 0 | 0 | 0 | 0 | 0210, 0216 | | custom | 28 | 28 | 0 | 0 | 0 | 0 | 0 | 0 | | | errors | 97 | 93 | 0 | 2 | 2 | 0 | 0 | 0 | 0036, 0213 | -| filters | 308 | 302 | 4 | 2 | 0 | 0 | 0 | 0 | 0101, 0178, 0203, 0215 | +| filters | 309 | 303 | 4 | 2 | 0 | 0 | 0 | 0 | 0101, 0178, 0203, 0215 | | fuzz | 32 | 32 | 0 | 0 | 0 | 0 | 0 | 0 | | | globals | 74 | 73 | 1 | 0 | 0 | 0 | 0 | 0 | 0212 | | i18n | 54 | 54 | 0 | 0 | 0 | 0 | 0 | 0 | | @@ -37,7 +37,7 @@ Snapshot of `python3 test/parity/generate.py --report` (Jinja2 3.1.6, Oct 2026): | tests | 58 | 57 | 0 | 1 | 0 | 0 | 0 | 0 | 0101 | | undefined | 106 | 106 | 0 | 0 | 0 | 0 | 0 | 0 | | | whitespace | 54 | 54 | 0 | 0 | 0 | 0 | 0 | 0 | | -| **total** | **1938** | **1894** | **25** | **15** | **4** | **0** | **0** | **0** | | +| **total** | **1939** | **1895** | **25** | **15** | **4** | **0** | **0** | **0** | | *output*: both render, text differs. *rejects*: C++ errors on a valid template. *accepts*: C++ renders a template Jinja2 rejects. *unsupported*: needs an Environment diff --git a/docs/tasks/0269-public-big-int-value.md b/docs/tasks/0269-public-big-int-value.md index 4382b2ec..da816106 100644 --- a/docs/tasks/0269-public-big-int-value.md +++ b/docs/tasks/0269-public-big-int-value.md @@ -1,5 +1,5 @@ --- -status: open +status: done priority: high area: api depends: [0172, 0178] @@ -23,3 +23,5 @@ that enters a template raises OverflowError. Coordinate with 0250's phase 0253, rows), abidiff shows `sizeof(Value)` unchanged, and docs/api-2.0.md documents `BigInt`. Plan: [docs/design/0178-big-integer-plan.md](../design/0178-big-integer-plan.md). + +Done in #596. diff --git a/docs/tasks/README.md b/docs/tasks/README.md index 65a49267..b107d187 100644 --- a/docs/tasks/README.md +++ b/docs/tasks/README.md @@ -300,7 +300,7 @@ files go under `touches` (0012 and 0024 each rewrite about a hundred rows). | [0230](0230-boost-free-core.md) | A Boost-free core | build | medium | open | | [0231](0231-robin-hood-successor.md) | robin_hood's upstream is archived | perf | high | open | | [0232](0232-dependency-pin-drift.md) | Dependency pins disagree across modes and some are mutable | build | low | open | -| [0233](0233-cpp-only-filter-stubs.md) | C++-only filters `camelize`, `underscorize`, `escapecpp`, `toxml`, `toyaml` do nothing | api | medium | open | +| [0233](0233-cpp-only-filter-stubs.md) | C++-only filters `camelize`, `underscorize`, `escapecpp`, `toxml`, `toyaml` do nothing | api | medium | done | | [0234](0234-github-tracker-cleanup.md) | Close GitHub issues that master already fixes, before 2.0 | release | medium | open | | [0235](0235-parity-feature-rows-drift.md) | `docs/parity.md` feature rows contradict master | parity | medium | open | | [0236](0236-meta-find-undeclared-variables.md) | Template introspection: `meta.find_undeclared_variables`, `Template.module` | api | low | open | @@ -334,7 +334,7 @@ files go under `touches` (0012 and 0024 each rewrite about a hundred rows). | [0266](0266-confined-filesystem.md) | `ConfinedFileSystem`: a loader that never leaves its root | api | medium | open | | [0267](0267-untrusted-mode-fuzzing.md) | Fuzz targets for templates written by untrusted users | robustness | high | open | | [0268](0268-untrusted-mode-followups.md) | Untrusted mode: additive follow-ups after the freeze | api | low | open | -| [0269](0269-public-big-int-value.md) | Public `BigInt` value (0178 phase 1, G1) | api | high | open | +| [0269](0269-public-big-int-value.md) | Public `BigInt` value (0178 phase 1, G1) | api | high | done | | [0270](0270-big-int-core-text-form.md) | Big integers in the core, text form (0178 phase 2) | parity | medium | open | | [0271](0271-big-int-backend-interface.md) | Big-integer backend interface and overflow promotion (0178 phase 3) | api | medium | open | | [0272](0272-big-int-boost-backend.md) | Boost.Multiprecision backend and `JINJA2CPP_WITH_BIGINT` (0178 phase 4) | build | medium | open |