Part 1 of my series on contributing structured serialization to the SSH layer of PcapPlusPlus.

How to read this post. Think of the call stack from your DSA class. Every time we ask a new question, we push a frame and dive deeper. Every time we find an answer, we pop back up to the question that sent us down. By the end, the stack is empty and the main question is answered. Watch for the markers: ⬇ means we are diving, ⬆ means we are returning.

Level 0: The main question

What is serialization?

Here is the one-sentence definition, which we will earn properly by the end:

Serialization is converting an in-memory data structure into a flat, self-contained format that can be stored, transmitted, or inspected outside the running program. Deserialization is the reverse: rebuilding the structure from that format.

That sentence hides at least four unanswered questions. Flat compared to what? Self-contained from what? What is a format? And why can't a program just hand over its memory?

Let's push our first frame.

⬇ Level 1: Why can't we just copy the memory?

Here is some data in a C++ program:

struct Person
{
    std::string name;
    int age;
};

Person p{"Bruce Wayne of Gotham City, heir to Wayne Enterprises", 35};

p is an object sitting in RAM. The tempting idea: "It's just bytes. I'll copy sizeof(Person) bytes and send them to another program."

It fails. To see why, run this experiment and look at the addresses:

std::cout << "sizeof(Person)         = " << sizeof(Person) << " bytes\n";
std::cout << "name length            = " << p.name.size() << " characters\n";
std::cout << "address of p (stack)   = " << static_cast<const void*>(&p) << "\n";
std::cout << "address of the text    = " << static_cast<const void*>(p.name.data()) << "\n";

My real output:

=== Experiment 1: where does the data live? ===
sizeof(Person)         = 28 bytes
name length            = 53 characters
address of p (stack)   = 0x61fecc
address of the text    = 0x939ad0

Notice that sizeof(Person) is a small, fixed number, yet the name has 53 characters. The characters cannot be inside the object. And look at the two addresses: they are in completely different regions of memory. The object p holds the text somewhere else.

A surprise from running it on two machines. On my Windows build sizeof(Person) printed 28. The same program on a 64-bit Linux machine printed 40. The code did not change, so the layout of the same struct did. I checked my compiler with g++ -dumpmachine and it reported mingw32, a 32-bit target (note the short six-digit addresses). With this standard library, a pointer is 4 bytes there and a std::string is 24 bytes, so the struct is 24 + 4 = 28. On a 64-bit build a pointer is 8 bytes, a std::string is 32 bytes, and the compiler pads the struct to a multiple of 8, giving 40. The lesson: the raw bytes of an object depend on the compiler and the machine, so you cannot ship them to another system and expect them to mean the same thing.

Where does the text live, then? That is a question for the next level.

⬇⬇ Level 2: Where does the data live? Stack vs heap

A running program uses two main regions of RAM.

The stack is the same LIFO stack you know from DSA, applied to memory: each function call pushes a stack frame holding its local variables, and returning pops it. That is why local variables vanish when a function ends.

The heap is for two situations: the size is unknown until runtime (a string could be 5 or 5 million characters), or the data must outlive the function that made it.

Both live in the same RAM, as two regions of one building with two floors. And a warning about names: this "heap" has nothing to do with the min-heap or max-heap data structure. It is an unfortunate shared word.

So a std::string is a small object on the stack holding a pointer, a length and a capacity. The actual characters live in a heap block it manages.

A footnote worth a footnote. Short strings may be stored inside the string object itself (the "small string optimization"). That is why the demo uses a deliberately long name: it forces the text onto the heap so we can see the effect.

⬆ Back at Level 1: so what does p actually contain?

It contains a pointer: an address. Which raises the next question.

⬇⬇ Level 2: Why is an address useless to anyone else?

Every process gets its own virtual address space. An address such as 0x5569db7512b0 is a number meaningful only inside the process that owns it, like a house number that exists only in one town. Copy the bytes of p into another program and you copy the house number, not the house. The other process finds something unrelated at that address, or nothing, and reading it may crash. This is a shallow copy: it duplicates the pointer but not the thing pointed to.

Beyond pointers, raw object bytes also depend on the compiler, the CPU, and the standard library: sizeof(std::string), padding between members, and byte order can all differ between machines.

⬆ Back at Level 1: the way out

If memory can't leave the process, we have to build something that can. We follow the pointers, collect the real data, and write it down in a flat form that has no addresses and no assumptions about layout. Here is the smallest possible serializer:

std::string toJson(const Person& p)
{
    return "{\"name\": \"" + p.name + "\", \"age\": " + std::to_string(p.age) + "}";
}

My real output:

=== Experiment 2: serialize to a flat text format ===
{"name": "Bruce Wayne of Gotham City, heir to Wayne Enterprises", "age": 35}

That text contains the real characters and the real number. It can be saved to a file, sent over a network, or read by a Python or JavaScript program. That is serialization.

Honest caveat. My toJson is a toy. A real one must escape special characters: a name containing a quote (") would produce broken JSON. Real serializers handle that for you, which is one reason we use a framework instead of string concatenation.

Level 1 is answered. But a skeptic asks: "Do we really need text? Can't we send raw bytes when the data is simple?" Yes, and that opens the deepest part of the dive.

⬇ Level 1: When are raw bytes fine?

Raw binary is compact and fast. It works when two conditions hold:

1. The data contains no pointers, only plain values such as fixed-size integers (uint32_t is always exactly 4 bytes).
2. Both sides agree on the exact layout, including byte order.

This clears up a common confusion: JSON versus raw binary is not "serialization versus no serialization." Both are serialization. They are just different formats.

And here a new question appears, because bytes alone say nothing.

⬇⬇ Level 2: What does a byte sequence mean?

Look at these four bytes: 00 00 01 2C.

• If both sides agree "this is a 4-byte unsigned integer", it means 300.
• If they agree "these are four separate flags", it means something entirely different.

Raw bytes have no meaning on their own. Meaning comes from an agreed rule, and that rule is the format.

How does 300 become 00 00 01 2C? One byte only holds 0 to 255, so we split the number in base 256 (like decimal digits, but with 256 symbols):

• 300 = 1 × 256 + 44
• The two "digits" are 1 and 44, which in hexadecimal are 01 and 2C
• Padded to four bytes: 00 00 01 2C

The receiver never sees "300". It sees four bytes and applies the agreed rule in reverse. Notice that the rule included something subtle: which byte comes first?

⬇⬇⬇ Level 3: Byte order (endianness)

In the decimal number 4,205, the 4 is the most significant digit (it changes the value most) and the 5 is the least significant. Bytes work the same way: in 00 00 01 2C, the byte 01 carries more weight than 2C.

Endianness is simply the decision of which byte to store first.

Why do both exist? Largely history: hardware designers chose differently and neither side won. Big-endian reads naturally in a hex dump; little-endian suits some arithmetic hardware. The early Internet standardized on big-endian.

Let's see it on a real machine:

std::uint32_t n = 300;
const unsigned char* mem = reinterpret_cast<const unsigned char*>(&n);

for (int i = 0; i < 4; ++i)
    std::printf("%02x ", mem[i]);          // bytes as MY machine stores them

unsigned char wire[4];
wire[0] = static_cast<unsigned char>((n >> 24) & 0xFF);
wire[1] = static_cast<unsigned char>((n >> 16) & 0xFF);
wire[2] = static_cast<unsigned char>((n >> 8)  & 0xFF);
wire[3] = static_cast<unsigned char>( n        & 0xFF);

for (int i = 0; i < 4; ++i)
    std::printf("%02x ", wire[i]);         // bytes in big-endian (network) order

My real output:

=== Experiment 3: byte order of 300 ===
bytes in MY machine's memory: 2c 01 00 00 
bytes in big-endian (network) order: 00 00 01 2c 

The first line is my machine's own little-endian order, and the second is the big-endian network order. Same number, two byte sequences.

Why does this matter if it never affects my own PC? Inside one machine, every instruction uses the same order and you never see the bytes. Byte order matters only when bytes cross a boundary: a network, a file, another machine. A little-endian machine that reads 00 00 01 2C straight into an integer sees 2C 01 00 00 in its own order, which is 738,262,272 instead of 300. That is why networking code converts ("network to host", ntohl and friends), and why documentation for every field in a protocol library should state its byte order.

(JSON text has no byte-order problem for numbers: the digits 3, 0, 0 are written in reading order. Byte order is a binary-format concern.)

⬆⬆ Back at Level 2: two things settled

Bytes mean what the format says, and the format includes the byte order. Which brings us to the question that has been waiting since Level 1.

⬇⬇ Level 2: How do two programs agree on a format?

They don't negotiate at runtime. They agree in advance, on paper. The agreement is a specification, and both programs are written to follow it.

For the Internet, many specifications live in documents called RFCs: Request for Comments. For SSH, the transport protocol is defined in RFC 4253, The Secure Shell (SSH) Transport Layer Protocol. The OpenSSH developers wrote their sender to follow it; the PcapPlusPlus developers wrote their reader to follow it. They never needed to speak to each other. If someone disagrees with the document, the symptom is garbage values or crashes.

SSH's basic data types are defined in a sibling document, RFC 4251 (The Secure Shell (SSH) Protocol Architecture, section 5). There, a uint32 is defined as a 32-bit unsigned integer stored as four bytes in order of decreasing significance, which it calls network byte order: big-endian. A string is a uint32 length followed by that many bytes, and a name-list is a string of comma-separated names. Specifications are often split across several RFCs like this.

⬇⬇⬇ Level 3: So what do SSH bytes look like?

RFC 4253 defines, among other things:

• The identification string. A text line at the start of a connection, in the form SSH-protoversion-softwareversion, optionally followed by a comment, and ended by carriage return and line feed.
• The binary packet. After that, messages travel as: a 4-byte packet length (uint32), a 1-byte padding length, the payload, random padding, and a MAC whose size depends on the negotiated algorithm (it is empty at the start, before any MAC has been agreed). Since uint32 is big-endian by RFC 4251, so is the packet length.
• Key exchange. The handshake in which the two sides agree on encryption algorithms. After keys are set up, the payload is encrypted and becomes opaque to anyone watching the wire.

Here is the insight that connects everything: those SSH wire bytes are already serialized. The sender took its in-memory data and wrote it into this exact binary layout. That was serialization. Anyone reading the capture must then deserialize it, which is what a parsing library does.

⬆⬆⬆ Back to Level 1: the stack is shrinking

Raw bytes are fine, but only because a format (written in a specification) gives them meaning. Serialization is not opposed to sending bytes. Sending bytes is serialization, whenever the bytes follow an agreed format.

Side quests: three things that sound similar but aren't

While learning this, you will meet other packet topics. They are neighbours, not synonyms:

Data can fail in any one of these ways while the other two are fine: perfect format but corrupted in transit; intact but never delivered; delivered but read in the wrong byte order. (A checksum is also just another field in a header, with its own size and byte order, so it is itself serialized.)

⬆ Back to Level 0: the stack is empty, the answer stands

Now the definition from the top of this post should read very differently:

Serialization converts an in-memory data structure (full of pointers and machine-specific layout) into a flat, self-contained format (no addresses, agreed meaning, agreed byte order) that can be stored, transmitted, or inspected outside the running program.

Four words to keep:

• Serializer: the code that writes the format (toJson above).
• Format: the agreed rules for the output (JSON, XML, YAML, or a binary layout like SSH's).
• Byte order: one rule inside a binary format.
• Round trip: serialize, then deserialize, and you get back the same data.

Why any of this matters for PcapPlusPlus

PcapPlusPlus is a C++ library for capturing, parsing and crafting network packets. For protocols like SSH it acts as an observer: it reads the already-serialized bytes from the wire and decodes them. It is the parsing (deserialization) half.

Its maintainers have added a generic serialization framework (PR #2259) so that any layer's decoded fields can be exported in a structured format. The framework adds an ISerializer interface, scopes for nested objects and arrays, and a JSON serializer. Each protocol layer then declares its fields once, and the serializer decides the output format. The roll-out is tracked in Issue #2278, where the SSH layer is one of the layers still waiting for support. That is the piece I am working toward.

To contribute it properly, I need to understand three more things, and each gets its own post:

1. How PcapPlusPlus represents a packet: layers as views over a raw buffer.
2. The serialization framework itself: fields, descriptors, and the serialize function.
3. The SSH layer's fields: what is honestly available to export, and what is hidden by encryption.

Next up: Part 2, "A Packet Is a Stack of Views."

Try it yourself

The three experiments above live in one small file. Build and run:

g++ -std=c++17 -Wall -Wextra -o serialization_demo serialization_demo.cpp
./serialization_demo

Your addresses and your sizeof(Person) in Experiment 1 may differ from mine, and that is itself the lesson: addresses belong to the process, and layouts belong to the compiler and machine, not to the data.

Where it started

CyberCube began as our major project. We could have built another full-stack app, but we wanted to try something different, so we moved into cybersecurity as an experiment. This post covers what we built and what its numbers do and do not mean.

From packets to alerts

Traffic can enter the platform in two ways:

• PCAP upload. You drag a packet-capture file into the web app. A parser I wrote unpacks the binary data field by field, reading the Ethernet, IP and TCP headers by hand. It then groups packets into two-way flows and runs the model on each one.
• Live capture. The sniffer tries to capture packets directly from the machine's network interface. That needs administrator privileges. When they are missing, it falls back to generated traffic so the live view keeps working. That fallback is meant for demos, not as real capture.

Detected threats appear on a React dashboard through a WebSocket connection (/ws/threats), so new alerts arrive without refreshing the page. The dashboard includes alert tables and a global threat map that plots where attackers appear to come from.

The dataset, and an honest caveat

Training a detection model needs labelled traffic. I generated a dataset of 12,000 records with 22 features, designed to follow the CIC-IDS2017 schema, and included attacker locations and MITRE ATT&CK labels. The data is synthetic. That means the accuracy numbers below show the pipeline works from end to end, but they are not a claim about performance on real-world traffic.

Choosing a model

I needed a classifier that is accurate, fast, and explainable, because analysts will only act on alerts they trust. I benchmarked four on an 80/20 split of the data (2,400 test samples):

• Gradient Boosting matched the accuracy but was about 1.8 times slower, roughly 30 microseconds per prediction against 17.
• The neural network was faster but less accurate, and hard to explain.
• Logistic Regression was the fastest but the least accurate.
• Random Forest gave the best balance.

I deployed Random Forest: about 99.5% accuracy, 99.37% macro-F1, around 17 microseconds per prediction, and explanations that show why each flow was flagged.

Explaining every alert

A label like "DDoS, 98% confidence" is not enough for an analyst, so each alert comes with more:

• SHAP waterfall charts. SHAP shows which features of a flow, such as the timing between packets or the count of TCP flags, pushed the model toward "attack" or "benign".
• MITRE ATT&CK mapping. Each threat type is tagged with a standard technique ID. For example, DDoS maps to T1498, port scans to T1046, brute force to T1110, web attacks to T1190 and botnet traffic to T1071.
• Plain-language summaries. The Gemini API turns an alert into a short root-cause explanation, sent through a backend proxy so the key stays on the server.

From detection to response

CyberCube also helps you act:

• Firewall rules. One click generates commands to block an attacker's IP: netsh advfirewall for Windows, iptables for Linux, and rules for Suricata and Snort.
• PDF incident reports. A report contains the severity, attacker location, MITRE technique, all 22 flow metrics, the SHAP-based root cause, and a space for the analyst's signature.

Multi-tenancy and deployment

Several organizations can use the same platform without seeing each other's data. Users sign in with Google OAuth, and every threat record carries a tenant key derived from the user's email using HMAC. Session tokens are signed and expire after 12 hours, and the WebSocket only delivers alerts that match the connected user's key.

The backend is FastAPI with SQLAlchemy, using SQLite by default and PostgreSQL (Neon) in production. A script injects sampled flows from the dataset into the system on a schedule, run by GitHub Actions, which keeps the dashboard populated. The app is deployed on Render and Vercel.

What I would improve next

• Train and test on real CIC-IDS2017 data and compare it with the synthetic results.
• Test the live sniffer on real network traffic instead of relying on the demo fallback.
• Add more attack types beyond the five currently mapped.
• Write automated tests for the PCAP parser and the prediction pipeline.

Takeaways

A detection model is only useful if an analyst can trust it and act on its output. The explanations, the standard threat labels and the ready-made firewall rules were as much of the project as the classifier itself. And being clear about what synthetic data can and cannot prove makes the results more believable.

• Live app: ai-powered-cyber-threat-detection-soc.onrender.com
• Code: github.com/NartanNaik/CyberCube
• Stack: Python, Scikit-learn, FastAPI, React, PostgreSQL, SHAP, Gemini API

The problem: a codebase does not fit in a prompt

The obvious fix is to send the whole repository to an AI model. That fails quickly. On a large raw repository, the input came to about 6.45 million tokens, far beyond any model's context window. My early attempts with a big project like CyberCube ended with an AI telling me it was too huge and to try a smaller one.

That is not a useful answer for a tool whose job is to understand big projects. So the real problem became deciding what the model should never see.

The scanner: cutting 98% of the noise

Most of a repository is not worth reading. Build artifacts, binaries, lockfiles and generated files add size and no understanding. I wrote a custom scanner in Python that:

• Skips ignored directories, file names and extensions (build output, binaries, lockfiles).
• Enforces size caps on individual files, on the raw upload and on the final text sent to the model.
• Ranks files by importance so the most useful code is kept when the cap is reached.

That is a 98% reduction in model input, and it removes the context-window failures.

Three ways to bring in code

• Folder picker. The browser reads the folder directly. The files are filtered on the client side first, so less data is uploaded, and then zipped with JSZip.
• ZIP upload. The archive is opened in the browser, filtered, and zipped again.
• GitHub URL. The backend fetches a public repository, up to a size limit, and processes it the same way.

Four depths of analysis

Not every question needs a 20-page report, so Unpack offers four analysis depths. Each uses its own prompt and token budget, and the backend picks a model to match:

1. Quick Summary: a short overview of what the project is.
2. Standard Architecture: the main components and how they connect.
3. Deep Dive: everything: stack, data flow, schema, file breakdown.
4. Diagnose: a full audit that also lists security and technical-debt issues.

Streaming the answer

A detailed analysis takes a while to generate, so the backend streams it back as newline-delimited JSON events instead of making you wait for one big response. There are four event types: delta carries new text, ping shows the model is still working, done returns the final report, and error reports failures. On the frontend, a small XMLHttpRequest helper tracks both the upload and the streaming response, so the user sees live progress.

Accounts and cost control

Users sign in with Google or GitHub through Supabase Auth. The first time someone logs in, the backend validates their session, creates an OpenRouter key for that user capped at $0, and stores it in Supabase along with a SHA-256 hash of the key. The key is tied to the user's email, so each person gets their own.

A $0 cap means free models only, so the app cannot produce a surprise bill, and one person's usage cannot use up another's allowance.

Users can also choose between two modes: Temp, where the analysis is not saved, and Persistent, where it is stored in a history table in Supabase so they can come back to it.

Exporting in the browser

Reports can be downloaded as PDF, DOCX or PPTX. All three are generated entirely in the browser, with jsPDF, docx and pptxgenjs, so nothing goes through the server. Every export has the same branding: a purple and gold watermark, an UNPACK footer and page numbers.

Nothing in this project was dramatically hard, but the PDF was the fiddliest, and generating the watermark for all three formats took more time than I expected.

Does it work?

Yes, and I can prove it with my own projects. The architecture notes I keep for CreatorLens AI and CyberCube were generated by Unpack. They include the system architecture, tech stack, database schema, features, file breakdown and a list of technical-debt issues, and that is exactly what I built it to produce.

What I would improve next

• Add rate limiting to the key-provisioning and analysis endpoints.
• Redact secrets such as API keys from source files before they reach the model.
• Add automated tests, starting with the scanner's filtering rules.
• Validate uploaded archives more strictly, beyond checking their size.

Takeaways

With LLM apps, deciding what not to send matters as much as the prompt. Filtering the input removed 98% of the tokens and made large repositories workable, and it came from understanding what a model actually needs to see to describe a project.

• Live app: unpack-tau.vercel.app
• Code: github.com/NartanNaik/Unpack
• Stack: React, FastAPI, Python, Supabase, OpenRouter

Step one: pasting usernames by hand

I started by pasting each creator's username into the app one at a time. That worked for ten profiles and became painful at a hundred.

So I added mass upload: collect usernames in a list, then add them all at once. It was faster, but I was still filling in every attribute myself, and that took even longer than collecting the names.

Step two: letting AI fill in the details

I connected the Gemini API so I no longer had to type attributes like nationality by hand. The AI reads the categories I have defined and fills them in for each profile. The part I like most is that if I add a new column, the AI fills that one in too, so the schema is not hard-coded.

Step three: a mobile app with a share button

The workflow was still slow because I had to open the web app and enter things. So I built a Flutter mobile app with a built-in share target. When I find a creator's reel, I share it to my app the same way I would send a reel to a friend, and it lands in the website's queue. The app signs in with Google, stores tokens securely on the phone, and sends the URL to the REST API with a bearer token.

It was better, but each share still took more than 5 seconds from tapping share until the app closed itself.

Step four: the Garbage Eater

While working on the app I had been saving reels, and I realised those saved reels were already a list of everything I wanted to add. So I used a scraper to export all my saved reels as a JSON file, then built a feature to paste that JSON in and import everything at once. I named it the Garbage Eater, because I throw a big messy file at it and it sorts everything out.

Now my whole job is to save a reel and keep scrolling. Everything else happens afterwards, with each reel taking under a second in my own use.

CreatorLens now has four ways to add profiles:

The staging pipeline

Raw URLs do not go straight into the main database. They pass through a staging queue, which is a two-step pipeline:

1. Queue. A shared reel (URL, username, thumbnail, caption) is saved in a staging collection and the request returns immediately.
2. Process. A separate request enriches that item: it fetches metadata such as followers, likes, comments and views, and runs the AI step. After that, I approve the item and it moves into the main profiles collection.

Splitting it this way means no single request has to do everything, and it lets me review items before they become real profiles.

Hitting Vercel's 10-second limit

I first hit Vercel's 10-second function limit with the Garbage Eater. Small files were fine, but a large JSON file took too long to process in one request, and the request was cut off.

The fix was to break the work into batches. The importer now sends items in chunks of at most 50 per request, and the staging pipeline processes them separately, so each request stays well inside the limit. The batches are throttled as well, so a big import does not flood the database.

Protecting a free-tier database

I run on Firestore's free tier, so every read and write counts against a daily quota. Three things keep usage down:

• A Bloom filter is a small in-memory structure that answers "have I seen this username before?" without touching the database. It can say "probably yes" by mistake, but it never says "no" for something it has seen, which makes it a safe quick check. Because it skips a database read, it is faster and saves quota.
• Per-route rate limiting caps how often each group of endpoints can be called. For example, mass upload allows 60 requests a minute, bulk actions 30, and the video refresh job only 5.
• Chunked, batched writes group many writes into transactions instead of sending one write per profile.

The Bloom filter has a limit I should be honest about: it lives in memory on each running server instance and is not loaded from the database. After a restart, or on another instance, it starts empty, so it reduces duplicates and database reads but does not guarantee zero duplicates.

A security detail worth sharing: column names

Because users can create custom columns, those column names end up in database queries. Firestore treats a dot in a field name as a path into nested data, so a malicious name could write to places it should not reach. A validation step now allows only letters, numbers, underscores and hyphens in column names, and I added middleware that blocks NoSQL and dot-notation injection before any input reaches a query.

The rest of the API follows the same careful approach:

• Short-lived JWTs with refresh-token rotation, so a leaked token expires quickly.
• AES-256-GCM encryption for sensitive fields.
• Google OAuth sign-in with per-user data isolation, so no one can see another user's profiles.

What I would improve next

• Share duplicate checks across server instances, so deduplication survives restarts.
• Add automated tests. The project currently has none, and the importer and staging logic would benefit most.
• Stream-parse very large JSON files instead of loading them whole.
• Move long-running enrichment to a proper background job queue.

Takeaways

Most of this project was removing the slowest step, one step at a time: first typing, then pasting, then AI, then the phone, and finally the importer. A serverless time limit does not have to block you if you split the work into small, separate requests.

• Live app: createrlens.vercel.app
• Code: github.com/NartanNaik/CreaterLens
• Stack: React, Node.js, Express, Firebase Firestore, Flutter, Vercel, Gemini API