Conversation
ManagedChannelUtil built every gRPC channel with usePlaintext() and the module had no setting for transport security, so the URLs and their metadata, which can include cookies, always crossed the network in the clear, and a frontier which requires TLS could not be reached at all. Spout, StatusUpdaterBolt and QueueRegulatorBolt all create their channel there. The channel is now built with Grpc.newChannelBuilder and credentials read from the topology configuration. urlfrontier.tls.enabled switches to TLS channel credentials. The server certificate is checked against the PEM file in urlfrontier.tls.trust.cert.collection, or against the JVM trust store if that key is not set. urlfrontier.tls.client.cert.chain and urlfrontier.tls.client.private.key (with an optional urlfrontier.tls.client.private.key.password) add a client certificate for mutual TLS. Setting only one of the two, or naming a file which cannot be read, fails the component at startup with a message naming the key or the file. Plaintext stays the default so that existing deployments keep working; the INFO line logged when a plaintext channel is created names the key that enables TLS. The three components pass their configuration to createChannel. No new dependency is needed: the TLS credentials come from grpc-api and the transport from the grpc-netty-shaded jar the module already has. The new test covers the credentials built for each configuration and runs TLS and mutual TLS round trips against a local gRPC server with certificates generated at test time, as well as the failing cases of an untrusted server certificate, a plaintext client and a missing client certificate.
Contributor
|
@rzo1 is it even possible to run URLFrontier with TLS at the moment? if so, do you have any instructions on how to do so? |
Contributor
Author
|
No, the frontier server doesn't do TLS at the moment. It only listens in plaintext, so today you'd need a TLS terminating proxy in front of it. I've opened crawler-commons/url-frontier#219 to add TLS to the server, the channels between nodes and the CLI client. Once that's released, the settings in this PR connect to it directly, without a proxy. |
dpol1
approved these changes
Sep 23, 2026
dpol1
left a comment
dpol1
left a comment
Member
There was a problem hiding this comment.
LGTM. Two notes on the README, the first follows from the question above.
| ## Transport security | ||
|
|
||
| The gRPC channels to the frontier are plaintext unless TLS is enabled. Plaintext is kept as the | ||
| default so that existing deployments keep working. The channel carries the URLs and their metadata, so enable TLS whenever the |
Member
There was a problem hiding this comment.
can you add what you said above? URLFrontier 2.6 needs a TLS proxy in front of it, until crawler-commons/url-frontier#219 is released
| The server certificate must be valid for the host name in `urlfrontier.address` or | ||
| `urlfrontier.host`. Setting only one of `urlfrontier.tls.client.cert.chain` and | ||
| `urlfrontier.tls.client.private.key`, or pointing a key at a file which cannot be read, fails | ||
| the component at startup. The settings apply to `Spout`, `StatusUpdaterBolt` and |
Member
There was a problem hiding this comment.
a failed handshake doesn't fail the startup: Spout and StatusUpdaterBolt use withWaitForReady() without a deadline and just wait. worth a line here
… behaviour on a failed handshake
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2098.
The gRPC channels built in
ManagedChannelUtil(used by the Spout, StatusUpdaterBolt and QueueRegulatorBolt) can now use TLS. Plaintext stays the default.urlfrontier.tls.enabled: TLS on or off, default falseurlfrontier.tls.trust.cert.collection: PEM file of trusted certificates; the JVM trust store is used if not seturlfrontier.tls.client.cert.chain,urlfrontier.tls.client.private.keyand the optionalurlfrontier.tls.client.private.key.password: client certificate for mutual TLSA half-configured client certificate or an unreadable file fails the component at startup. No new dependencies. The tests cover TLS and mutual TLS round trips against a local gRPC server using certificates generated during the test.