py: modify kafka integration test to support multi variant test methods#5809
py: modify kafka integration test to support multi variant test methods#5809
Conversation
| # export SCHEMA_REGISTRY_URL= http://localhost:8081 | ||
|
|
||
|
|
||
| KAFKA_BOOTSTRAP = env( |
There was a problem hiding this comment.
KAFKA_BOOTSTRAP, SCHEMA_REGISTRY, and KAFKA_ADMIN are all evaluated at module load time (lines 47-56). Importing this file during pytest discovery or linting triggers live socket/HTTP connections to the CI Kafka broker and Schema Registry. A missing connection will fail the import and break any test run that includes this file, even for unrelated tests.
Consider lazy-initialising these (e.g., in setUpClass or via a module-level None populated on first use) or guard the connectivity checks behind an explicit flag/env var.
| TEST_CONFIGS = [ | ||
| {"id": 0, "limit": 10, "partitions": [0], "sync": False}, | ||
| { | ||
| "id": 1, |
There was a problem hiding this comment.
Variant 0 has "partitions": [0] and "sync": False but no "start_from". The old code passed "start_from": "earliest" unconditionally. sql_loopback_table only adds start_from if v.start_from is set, so variant 0 uses whatever the connector default is. If the default is "latest" the loopback table gets 0 rows and the hash validation fails. Please verify and add "start_from": "earliest" to variant 0 if needed.
|
When I run this locally, it fails because topics Since the topic names are hardcoded, I think that multiple runs at the same time are going to interfere with each other. |
blp
left a comment
There was a problem hiding this comment.
If you make the change we discussed, then this is fine, so I will approve it in anticipation of that.
- Introduces multiple test methods per pipeline variant. - Generates connector configurations from predefined variant settings. - Inherits from SharedTestPipeline instead of unittest.TestCase to allow single SQL compilation. - Separates SQL definitions into different functions instead of a single string. - Adds helper functions for polling and loopback validation. Signed-off-by: rivudhk <rivudhkr@gmail.com>
- Removed the SharedTestPipeline framework because the tests were accessing topics not assigned to them - Introduced automatic Kafka topic creation when needed. - Renamed the method 'extract_kafka_avro_artifacts' to 'extract_kafka_schema_artifacts' for clarity. - Renamed the method 'cleanup_kafka' to 'cleanup_kafka_schema_artifacts' for clarity. - Added TCP and HTTP checks to verify the availability of the Kafka broker and Schema Registry before tests run. - Added a RUN_ID mechanism, allowing individual test variants to be executed selectively. Signed-off-by: rivudhk <rivudhkr@gmail.com>
… connections - Replace run-id based topic names by unique UUID based topic names to avoid collisions across test runs - Lazy-initialize Kafka and Schema registry connections to prevent import-time network calls Signed-off-by: rivudhk <rivudhkr@gmail.com>
This PR modifies the Kafka Avro integration tests as follows:
extract_kafka_avro_artifactstoextract_kafka_schema_artifactscleanup_kafkatocleanup_kafka_schema_artifactsChecklist