Cloudpack rude questions and answers

Is the Webpack dev server experience really that broken?

Your mileage may vary!

We've used the webpack dev server experience for a very long time in many projects, and for many situations, it works fine. For large-scale enterprise web applications consisting of hundreds or thousands of packages and tens of thousands of source files, we see major scale issues, causing a reassessment of the situation.

The Outlook app consists of over 2600 unique packages and over 40,000 TypeScript and SASS files to build. It takes 7-10 minutes to just start the app in webpack-dev-server, and often that process fails because of loading too many files. When a developer edits a source file, sometimes it works, and sometimes updates just don't happen. There is no live-reloading let alone hot-reloading. This isn't an acceptable inner-loop for an Outlook developer, yet alone any of the shared package developers who want to try their new bits in Outlook.

So... every project has their own unique webpack config. How will Cloudpack work with every app?

The fact that every app in our organization has their own unique webpack config is a problem in itself. Cloudpack might not solve every app scenario, but if we can make it easy to standardize on good things and hard (but not impossible) to deviate, this leads towards the pit of success.

There are apps and there are libraries. Let's start with libraries first and then talk about apps.

To produce a browser-consumable standalone library, we need some conformance:

• Must expose an entry-point (ideally via exports in package.json, with node condition for Node-specific entries)
• Static resources are in a prescriptive location, clearly identified and in a supported format
• Must follow strict Semantic Versioning practices. Breaking API changes, such as removing exported names, require a major version bump
• Follow package export best practices

Cloudpack aims to provide a straightforward solution for each of these. However the author can deviate and use their own solutions via bundler capabilities.

Apps have a superset requirement of libraries; there are one or more entry points that corresponds with HTML entry points. The HTML pages may be automatically generated or generated from a template.

Where things get complicated is when the HTML page must live on a particular domain in order for auth to work. When the page can't be hosted by the local dev server, it pushes the requirement for the import map to be injected by the server. This is supported with a bootstrap route.

Is cross-repo development really something we need to worry about? Won't one repo solve this?

Giant monorepos solve many things related to symlinking dependencies together, but there are many projects which can't afford this. Any time part of the project cross the private/public boundary, the repo must follow. Some code such as the component library may live in a public GitHub repo, while other code such as the app code may only exist in private Azure DevOps.

We need solutions that scale to many repos and allow for a better experience than npm link, including properly accounting for nested dependencies.

How will solving the inner loop increase testing?

Today, it is just not discoverable how to test your changes.

Even when you know which apps consume your library, you need to understand how to get their inner loops working.

Once you've figured that out, you need to link your package to their setup. Due to differences in package managers, this doesn't always work.

Even with the same package managers, this doesn't always work - sometimes extra npm dependencies are brought in due to reliance on symlinking, when they should have been de-duped. You end up getting errors in the browser about duplicate React versions and such.

Cloudpack aims to make it easy to get an inner loop going for your scenario in any app. This is still a work in progress (cloudpack link is implemented for local libraries and apps already), but in the future it might look something like:

• Go into some library source.
• Run cloudpack
• A dashboard opens up, listening known apps that use the library.
• Click one of the apps to start an inner loop and work on your library in the context of an app you may not even have locally cloned. (Of course, you can clone it.)

How can this work? Essentially, we need any package or collection of packages in any app to be side-loadable.

• Apps need to load in a predictable manner and be "hosted" through local express, or through a service which allows an import map to be included on the page.
• The import map can be stored in a service and fetched.
• As long as we can deliver you a URL to test an app with a given import map and appropriate CORS settings, we can host the client code you care about from localhost while all the other code comes from a remote location.

Are there other tools we should be using instead of building our own?

There are a number of tools that are similar to the Cloudpack idea:

• Vite - An inner-loop build tool which separates bundled externals from individual file-load internals. Uses esbuild/rollup under the hood. Vite is the closest related tool to the idea of Cloudpack, but previous investigations ruled it out for use at scale.
• Snowpack - Snowpack is also similar to Cloudpack; externals are provided by a bundle service (Skypack CDN). The primary issue with the Skypack bundles are that while they are browser consumable, they hard-code dependency versions, meaning the app can't change those versions using an import map.

This web tool ecosystem evolves fast; by the time this text is read, 3 new tools will be likely be competing for viability and improvements in various things.

This is why Cloudpack is primarily looking to abstract existing work. The less the config and API surface we have, the easier it is to change out underlying approaches.

Additionally, as a future goal, Cloudpack aims to offer some UX around managing your app. With a dashboard UI surface, we can provide a better experience than the command line for detecting dependency issues, compilation issues, and bundle size issues.

Is there a mode you could use Webpack to produce bundles under the hood then?

Cloudpack supports using Webpack or rspack (with ESM output) as the underlying bundler for a specific package as a bail-out for certain cases not supported by other bundlers. It's also used to produce production mode bundles as of writing. However, Webpack is far too slow to be the default choice of bundler.

Inner loop - will the "load per-package bundles" approach scale?

In testing so far, this approach has worked for local development including in large projects, but HTTP/1.1's limitations remain somewhat of a performance bottleneck. We plan to investigate HTTP/2 or HTTP/3 once support in browsers and Node libraries is good enough. We may also need to produce "priming" files which parallelize the download to remove import waterfalls.

Another bottleneck is packages that use many deep import paths, each of which must be handled as a separate bundle. We recommend that repos work towards minimizing deep imports and properly using exports maps.

What about outer loop?

We have ideas for shipping and services features we'd like to investigate at some point, but due to being a small team, for now we're focusing on implementing a great inner dev loop experience and improving ability to test changes between repos. We're also considering how Cloudpack could be used for better production bundling.

Do you collect telemetry?

Telemetry is only collected if a telemetry config is provided, and it's only sent to the specified Application Insights instance. For teams within Microsoft, the Cloudpack team provides a shared instrumentation key to track usage on our internal dashboards

Once you build it, what's the plan to support it?

Cloudpack is implemented by a small team in Office, and we're also available to support partner teams. Due to limited resources, we're holding off implementing any features that would require us to run and support a cloud service. (All current cloud-based features such as sync and telemetry are set up so you bring your own Azure storage.)

Where can I learn more about what's happening with it?

See the status page.

How can I help?

See contributing info.