Fast Takeoff News

A Next.js application aggregating executive orders from the Federal Register API and generating real-time reports from Discord channel messages. Written in TypeScript, it uses Tailwind CSS for styling, Radix UI for components, and deploys via Cloudflare Workers with KV caching. The worker extends OpenNext with scheduled() and queue() handlers for cron-triggered report generation and financial data processing.

• Purpose: Fetch and display executive orders, monitor Discord channels for bot activity and generate structured reports using a configurable AI provider (currently Gemini 2.5 Flash via OpenRouter, with Groq/Llama 4 Maverick as alternative).
• Deployment: Hosted at news.fasttakeoff.org via Cloudflare Workers.
• Key Integrations: Federal Register API, Discord API, configurable AI Provider (e.g., Groq, OpenRouter), Clerk for authentication, Stripe for subscriptions, Twitter API, Instagram Graph API, Facebook Graph API.

• Framework: Next.js 15.2.4 (React 19.0.0)
• Language: TypeScript 5.x
• Styling: Tailwind CSS 3.4.14 with custom theming (src/app/globals.css), shadcn/ui components
• UI Primitives: Radix UI (@radix-ui/react-*) for accordion, dialog, select, etc. (often via shadcn/ui)
• Deployment: Cloudflare Workers (wrangler.toml), built with OpenNextJS (@opennextjs/cloudflare v1.11.0)
• Authentication: Clerk (@clerk/nextjs)
• Payments: Stripe (via API, for subscriptions)
• External APIs:
  • Federal Register: https://www.federalregister.gov/api/v1
  • Discord: https://discord.com/api/v10
  • AI Providers: (Configurable, see src/lib/config.ts)
    • Groq: https://api.groq.com/openai/v1
    • OpenRouter: https://openrouter.ai/api/v1
  • Twitter API: https://api.twitter.com/2/
  • Instagram Graph API: https://graph.instagram.com/
  • Facebook Graph API: https://graph.facebook.com/
  • Cloudflare Browser Rendering API: https://api.cloudflare.com/client/v4/accounts (for Instagram post image generation)
  • Cloudflare R2 Storage: https://images.fasttakeoff.org (for image hosting)
• Dependencies: groq-sdk, lucide-react, class-variance-authority, full list in package.json
• Configuration: ESLint (eslint.config.mjs), PostCSS (postcss.config.mjs), TypeScript (tsconfig.json)

All time durations are centralized in src/lib/config.ts under the TIME export. Avoid hardcoded multiplications like 24 * 60 * 60 or 6 * 60 * 60 * 1000.

• Constants (seconds): SECOND_SEC, MINUTE_SEC, HOUR_SEC, DAY_SEC, WEEK_SEC, MONTH_30_SEC, YEAR_365_SEC
• Constants (milliseconds): SECOND_MS, MINUTE_MS, HOUR_MS, DAY_MS, WEEK_MS, MONTH_30_MS, YEAR_365_MS
• Common windows (ms): FIVE_MINUTES_MS, FIFTEEN_MINUTES_MS, THIRTY_MINUTES_MS
• Helpers: minutesToMs(n), hoursToMs(n), daysToMs(n), minutesToSec(n), hoursToSec(n), daysToSec(n)
• Backward-compatible aliases: ONE_HOUR_MS, TWO_HOURS_MS, SIX_HOURS_MS, TWENTY_FOUR_HOURS_MS

Examples:

import { TIME } from "@/lib/config";

// 24h window in ms
const cutoff = Date.now() - TIME.DAY_MS;

// Cache for 10 minutes in seconds
await cache.put(ns, key, value, TIME.minutesToSec(10));

// Dynamic window
const lastXHours = Date.now() - TIME.hoursToMs(x);

The application's core data handling is managed by services and transformers within the src/lib/ directory:

• src/lib/data/channels-service.ts (ChannelsService): Fetches Discord channels from the API, filters them based on criteria (type, emoji prefix, permissions), and caches the results. Essential for identifying relevant channels for news monitoring.
• src/lib/data/messages-service.ts (MessagesService): Responsible for fetching messages from specified Discord channels, handling pagination, and caching them. Provides the raw source material for report generation.
• src/lib/data/executive-orders.ts: Fetches lists of executive orders and individual order details from the Federal Register API. Implements caching for individual orders and mechanisms to look up orders by number.
• src/lib/data/report-service.ts (ReportService): Core service that orchestrates the generation of news reports. It retrieves messages, prepares prompts for the AI, calls the AI provider, processes the response, and caches the generated reports. It also handles social media posting through integration with Twitter, Instagram, and Facebook services.
• src/lib/utils/report-ai.ts (ReportAI): Handles AI interactions for report generation, including prompt preparation, API calls, and response processing.
• src/lib/utils/report-cache-d1.ts (ReportCacheD1): Manages report caching operations using D1 database as the source of truth with KV for performance optimization, including storage, retrieval, and cleanup of generated reports.
• src/lib/transformers/executive-orders.ts: Transforms raw executive order data fetched from the Federal Register API into a structured ExecutiveOrder type used throughout the application, simplifying data handling in the frontend and other services.

• src/lib/utils.ts: Provides a collection of general utility functions for tasks such as date/time formatting and manipulation (e.g., formatDate, formatTime, getStartDate), parsing specific text formats (e.g., parseDispositionNotes for executive order relations), class name construction for Tailwind CSS (cn), and accessing the Cloudflare Worker environment context (getCacheContext).
• src/lib/api-utils.ts: Offers helpers for Next.js API routes, including a withErrorHandling higher-order function to standardize error responses and JSON formatting, and default cache headers (API_CACHE_HEADERS) for API responses.

The Current Events section provides real-time monitoring and AI-powered report generation from Discord channels.

• Monitors specific Discord channels for bot messages
• Filters messages using bot username and discriminator
• Dynamic Report Generation: AI-powered reports with headline, city, and body content
• Adaptive Window System: Reports generated based on channel activity patterns
• Translation support for multiple languages

The system uses activity-driven windows instead of fixed intervals for more responsive news coverage:

Channel Classification (7-day rolling average):

• High Activity (≥8 msgs/report): Generate when ≥3 messages OR after 30min max
• Medium Activity (3-7 msgs/report): Generate when ≥2 messages OR after 60min max
• Low Activity (<3 msgs/report): Generate when ≥1 message OR after 180min max

Overlap Prevention:

• Checks last 4 hours for existing reports
• Skips generation if new window would have ≥50% overlap with recent report
• Ensures fresh content in each report without duplication

Evaluation Schedule:

• Runs at 15 and 45 minutes past the hour via cron (15,45 * * * *)
• Split from MESSAGES cron to avoid subrequest quota limits (each gets 1,000 quota)
• Processes channels in batches of 3 to avoid system overload
• Uses bulk D1 queries (2 queries vs ~40-50) to minimize subrequests (~80% reduction)
• Stores evaluation metrics for monitoring dashboard

Example Scenarios:

🟡us-politics-live (20 avg msgs): 3 messages in 10min → Generate at 15min check
🔴ukraine-russia-live (12 avg msgs): 0 messages for 30min → Generate at 30min limit  
🟠myanmar (3 avg msgs): 1 message appears → Generate at next 15min check
🟡geological-events (3 avg msgs): Quiet for 180min → Generate when next event occurs

Benefits:

• Breaking news reports within 15-30 minutes vs 2+ hours
• ~30-40% fewer reports during quiet periods
• Natural alignment with actual news cycles
• Same AI prompts and quality, just smarter timing

• GET /api/reports - Fetch cached reports
• GET /api/translate - Translate report content

# Discord Integration
DISCORD_TOKEN=<your-discord-bot-token>
DISCORD_GUILD_ID=<your-guild-id>

# AI Provider (Choose one)
GROQ_API_KEY=<your-groq-api-key>
OPENROUTER_API_KEY=<your-openrouter-api-key>

• Verify Discord bot permissions and token
• Check AI provider status and API keys
• Review logs for specific error messages

Interactive 3D visualization of news reports with geographic data.

• 3D visualization using Three.js
• Real-time geocoding of news locations
• Interactive markers for report display

• GET /api/geocode - Geocode city names for globe placement

Automated aggregation and AI-powered summarization of Brazilian news.

• Aggregates news from major Brazilian RSS feeds
• Two-stage AI processing:
  • Curation based on impact and verifiability
  • Structured summarization by topics (Giro Geral, Mercado)
• Historical archive with timestamp selection
• Hourly updates with caching
• Topic selection tabs for different news focuses

• GET /api/summaries/list?topic=geral|mercado - List available summaries for a topic
• GET /api/summaries/[key] - Fetch specific summary

# Social Media Integration
INSTAGRAM_ACCESS_TOKEN=<your-instagram-long-lived-access-token>
TWITTER_CLIENT_ID=<your-twitter-app-client-id>
TWITTER_CLIENT_SECRET=<your-twitter-client-secret>
FACEBOOK_APP_ID=<your-facebook-app-id>
FACEBOOK_APP_SECRET=<your-facebook-app-secret>
FACEBOOK_PAGE_ID=<your-facebook-page-id>
FACEBOOK_PAGE_ACCESS_TOKEN=<your-facebook-page-access-token>

# Discord Integration
DISCORD_TOKEN=<your-discord-bot-token>
DISCORD_GUILD_ID=<your-guild-id>

• RSS Feed Issues:
  • System continues with available feeds if some fail
  • Check logs for specific feed errors
  • Verify source URLs in configuration
• Cache Management:
  • Clear KV cache through Cloudflare dashboard
  • Cache refreshes hourly
  • Force refresh with ?fresh=true parameter

Tracks and displays executive orders from the Federal Register API.

• Fetches and displays executive orders
• AI-powered summarization
• Historical tracking and caching
• Structured display of metadata and full text

• GET /api/executive-orders - List orders
• GET /api/executive-orders/[id] - Get specific order
• GET /api/summarize - Generate AI summary

AI-powered transparency system that maps content in generated news reports back to their original Discord source messages.

• Interactive Highlighting: Report text segments are visually highlighted and linked to source messages
• Confidence Scoring: AI provides confidence scores (0-1) for each attribution with visual feedback
• Rich Tooltips: Hover over highlighted text to see original message content, timestamps, and attribution confidence
• Multiple Text Matching: Uses exact, normalized, and fuzzy matching strategies to find text positions
• Intelligent Caching: Attributions are cached using Cloudflare KV with generation locks to prevent duplicate processing
• Batch Processing: Pre-generates attributions for multiple reports with configurable concurrency
• Graceful Degradation: Falls back to plain text display when attributions are unavailable

• AI Integration: Uses structured JSON schema responses from configurable AI providers (Groq/OpenRouter)
• Fuzzy Text Matching: Implements sophisticated position finding with 80% similarity threshold for sentence-level matching
• Error Resilience: Retry logic with exponential backoff, falls back to previous cached versions on failures
• Performance Optimization: Lazy loading, efficient paragraph processing, and color assignment based on message index

• GET /api/source-attribution?reportId=X&channelId=Y - Fetch source attributions for specific report

Key settings in src/lib/config.ts:

SOURCE_ATTRIBUTION: {
    MAX_ATTEMPTS: 3,              // Retry attempts for AI calls
    OUTPUT_BUFFER: 4096,          // Tokens reserved for AI output
    MAX_CONTEXT_TOKENS: 32000,    // Maximum context window
    SYSTEM_PROMPT: "...",         // AI system instructions
    PROMPT_TEMPLATE: "..."        // Template for user prompts
}

interface SourceAttribution {
  id: string; // Unique identifier
  startIndex: number; // Start position in report body
  endIndex: number; // End position in report body
  text: string; // Attributed text content
  sourceMessageId: string; // Source Discord message ID
  confidence: number; // AI confidence score (0-1)
}

interface ReportSourceAttribution {
  reportId: string; // Report identifier
  attributions: SourceAttribution[]; // Array of attributions
  generatedAt: string; // Generation timestamp
  version: string; // Attribution system version
}

• AttributedReportViewer: Main container managing attribution state and API calls
• InteractiveReportBody: Renders report text with highlighted attributed segments
• SourceTooltip: Rich tooltips displaying source message details and confidence scores

• Integrated with cron jobs for background attribution generation
• Uses same caching strategy as main reports (REPORTS_CACHE KV namespace)
• Cache keys: attribution:{reportId}
• Supports cache invalidation for specific reports

Aggregates multiple RSS sources (Brazil + global) into a single, sortable stream and exposes a simple UI and API.

• Pulls items from configured RSS sources in src/lib/config.ts → RSS_FEEDS.
• Global feeds added (examples): Bloomberg Markets/Economics, Axios, Yahoo Finance; Brazil sources already present.
• Region filtering via RSS_FEED_REGIONS mapping (BR, US) used by the aggregator/API and the UI toggle.
• Light caching (~3 minutes) in FEEDS_CACHE for the aggregated API.

• Page: /feeds — reverse‑chronological list with source label, time, snippet, thumbnail.
• Filters: All / US / Brazil — updates URL (?region=US|BR) without page reload.
• Discoverability: linked in the header as “Feeds”.

• GET /api/news/feeds
  • Query params:
    • feeds: optional CSV of feed IDs (exact names from RSS_FEEDS)
    • perFeedLimit: 1–50 (default 20)
    • limit: 1–200 overall (default 100)
    • region: US or BR (optional)
  • Response: array of items with { sourceId, sourceUrl, title, link, pubDate, contentSnippet?, enclosureUrl?, categories? } sorted by pubDate desc.

• Add feeds: edit src/lib/config.ts RSS_FEEDS.
• Classify region: edit RSS_FEED_REGIONS ('US' | 'BR').

• Unit: tests/unit/feeds-aggregator.test.ts checks sorting and region=BR filtering using mocks (no network).

# API Configuration
SERVER_API_URL=https://news.fasttakeoff.org  # Production URL
# Use http://localhost:8787 for local development

# Authentication & Payments
NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=<your-clerk-publishable-key>
CLERK_SECRET_KEY=<your-clerk-secret-key>
STRIPE_SECRET_KEY=<your-stripe-secret-key>
STRIPE_PRICE_ID=<your-stripe-price-id>
STRIPE_WEBHOOK_SECRET=<your-stripe-webhook-secret>

# Report Generation Testing
scripts/generate-reports.sh  # Generate test reports
scripts/check-truncation.sh  # Verify report truncation
scripts/full-test.sh        # Full test including build

# Local Development
bun run dev                 # Start local server
bun run preview:patch:test  # Test with Cloudflare Worker
bun run deploy             # Deploy to production

• All endpoints use stale-while-revalidate strategy
• Default cache TTL: 1 hour
• KV namespaces:
  • EXECUTIVE_ORDERS_CACHE
  • REPORTS_CACHE
  • CHANNELS_CACHE
  • FEEDS_CACHE
  • AUTH_TOKENS
  • GEOCODE_CACHE
  • SUBSCRIPTIONS_CACHE
• R2 buckets:
  • INSTAGRAM_IMAGES - Stores generated Instagram post images with 7-day retention

MIT

src/
├── app/                  # Next.js pages and API routes
│   ├── api/              # API endpoints
│   │   ├── channels/       # Discord channel related (e.g., /active)
│   │   ├── reports/        # Report generation and retrieval
│   │   ├── geocode/        # Geocoding city names for News Globe
│   │   └── stripe/         # Stripe checkout and webhook
│   ├── current-events/   # Discord channel monitoring UI
│   ├── executive-orders/ # Executive order display UI
│   ├── news-globe/       # Interactive 3D news globe UI
│   ├── profile/          # User profile page (Clerk integrated)
│   ├── sign-in/          # Clerk sign-in page
│   ├── sign-up/          # Clerk sign-up page
│   ├── privacy-policy/   # Privacy policy page
│   └── globals.css       # Tailwind CSS configuration
├── components/           # Reusable React components
│   ├── current-events/   # Channel-specific UI (e.g., ChannelCard.tsx)
│   ├── executive-orders/ # Executive Orders UI
│   ├── source-attribution/ # Source attribution components
│   │   ├── AttributedReportViewer.tsx  # Main attribution container
│   │   ├── InteractiveReportBody.tsx   # Highlighted report text
│   │   └── SourceTooltip.tsx           # Rich source tooltips
│   ├── NewsGlobe.tsx     # 3D interactive news globe component
│   ├── Header.tsx        # Application header with navigation and user auth
│   ├── Footer.tsx        # Application footer
│   └── ui/               # shadcn/ui components (e.g., button.tsx)
├── lib/                  # Utilities and data logic
│   ├── data/             # Core services
│   │   ├── channels-service.ts      # Discord channel management
│   │   ├── messages-service.ts      # Discord message handling
│   │   ├── report-service.ts        # Report generation orchestration
│   │   ├── executive-orders.ts      # Executive order data handling
│   │   ├── feeds-service.ts         # RSS feed processing
│   │   └── rss-service.ts           # RSS feed fetching
│   ├── utils/            # Utility services
│   │   ├── report-ai.ts             # AI integration for reports
│   │   ├── report-cache-d1.ts       # Report caching logic (D1 + KV)
│   │   ├── report-utils.ts          # Report-related utilities
│   │   ├── twitter-utils.ts         # Twitter-specific utilities
│   │   └── source-attribution/      # Source attribution system
│   │       ├── source-attribution-service.ts  # Attribution orchestration
│   │       ├── source-attribution-ai.ts       # AI-powered attribution generation
│   │       └── index.ts                       # Attribution exports
│   ├── transformers/     # Data transformation (e.g., executive-orders.ts)
│   ├── types/            # Domain-organized TypeScript interfaces
│   ├── instagram-service.ts         # Instagram API integration
│   ├── twitter-service.ts           # Twitter API integration
│   ├── facebook-service.ts          # Facebook API integration
│   ├── ai-config.ts                 # AI provider configuration
│   └── config.ts                    # Main application configuration
.gitignore              # Excludes node_modules, .next/, etc.
cloudflare-env.d.ts     # Cloudflare Workers env typings
package.json            # Dependencies and scripts
wrangler.toml           # Cloudflare Workers configuration

• Node.js 20+
• bun 1.0+ (recommended package manager)
• Cloudflare account with Workers and KV access
• Environment variables:
  • DISCORD_TOKEN: Discord bot token
  • DISCORD_GUILD_ID: Target Discord guild ID
  • GROQ_API_KEY: Groq API key for report generation (if Groq is active AI provider)
  • OPENROUTER_API_KEY: OpenRouter API key (if OpenRouter is active AI provider)
  • STRIPE_SECRET_KEY: Stripe secret key for payments
  • STRIPE_PRICE_ID: Stripe price ID for the subscription product
  • STRIPE_WEBHOOK_SECRET: Stripe webhook secret for verifying incoming events
  • NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY: Clerk publishable key
  • CLERK_SECRET_KEY: Clerk secret key
  • NEXT_PUBLIC_CLERK_SIGN_IN_URL=/sign-in
  • NEXT_PUBLIC_CLERK_SIGN_UP_URL=/sign-up
  • NEXT_PUBLIC_CLERK_AFTER_SIGN_IN_URL=/
  • NEXT_PUBLIC_CLERK_AFTER_SIGN_UP_URL=/

git clone https://github.com/ghsaboias/news.fasttakeoff.org.git
cd news.fasttakeoff.org
bun install

Create .env.local:

DISCORD_TOKEN=<your-discord-bot-token>
DISCORD_GUILD_ID=<your-guild-id>
GROQ_API_KEY=<your-groq-api-key>
OPENROUTER_API_KEY=<your-openrouter-api-key>
STRIPE_SECRET_KEY=<your-stripe-secret-key>
STRIPE_PRICE_ID=<your-stripe-price-id>
STRIPE_WEBHOOK_SECRET=<your-stripe-webhook-secret>
NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=<your-clerk-publishable-key>
CLERK_SECRET_KEY=<your-clerk-secret-key>
SERVER_API_URL=http://localhost:8787 # Or your deployed worker URL
TWITTER_CLIENT_ID=<your-twitter-client-id>
TWITTER_CLIENT_SECRET=<your-twitter-client-secret>
INSTAGRAM_ACCESS_TOKEN=<your-instagram-access-token>
# INSTAGRAM_ACCOUNT_ID=<your-instagram-account-id> # If moved to env var

Start the dev server:

bun run dev

Access at http://localhost:3000.

Build locally:

bun run build

Preview with Cloudflare Workers:

bun run preview

Deploy to Cloudflare:

bun run deploy

Generates .open-next/ artifacts and deploys via wrangler.

• /api/reports: Generates reports from Discord messages via configurable AI provider; GET fetches cached summaries.
• /api/channels: Retrieves guild channels.
• /api/geocode: Geocodes a city name (used by News Globe).
• /api/source-attribution: Fetches source attributions for specific reports, mapping report content to original Discord messages.
• /api/stripe/checkout: Initiates a Stripe checkout session for subscriptions.
• /api/stripe/webhook: Handles incoming Stripe webhook events (e.g., checkout.session.completed).
• /api/oembed/twitter: Provides Twitter OEmbed data for embedded tweets.
• /api/linkedin/auth: LinkedIn OAuth authentication.
• /api/linkedin/callback: LinkedIn OAuth callback handler.
• /api/linkedin/test: LinkedIn API testing endpoint.
• /api/messages/heatmap: Generates message activity heatmap data.
• /api/emails: Email handling and management.
• /api/prompt-test: AI prompt testing and validation.

• /api/rss/[feedId]: RSS feed for specific content categories.
• /sitemap.xml: Main sitemap for SEO.
• /sitemap-index.xml: Sitemap index for large sites.
• /news-sitemap.xml: News-specific sitemap for search engines.

• /current-events: Displays active channels with message previews and report generation.
• /executive-orders: Lists and details executive orders with pagination and search.
• /news-globe: Shows an interactive 3D globe with geolocated news reports.
• /brazil: Displays AI-generated summaries of Brazilian news, with historical archive access and topic selection tabs.
• /message-activity: Displays message activity heatmap for Discord channels.
• /power-network: Interactive network visualization of political and business relationships.
• /profile: User profile page, shows subscription status and allows users to subscribe.
• /sign-in: User sign-in page (Clerk).
• /sign-up: User sign-up page (Clerk).
• /privacy-policy: Application's privacy policy.
• /rss: RSS feed endpoint for syndication.
• /: Home page featuring an animated hero section, and previews of the latest news reports and executive orders. Server-rendered with initial data passed to client components (SEO-optimized).

The Brazil section provides AI-generated summaries of Brazilian news, with features for historical tracking, topic selection, and comparison:

• Main View (/brazil): Shows the latest AI-generated summary of Brazilian news, with a dropdown selector for accessing historical summaries and tabs for topic selection (e.g., Giro Geral, Mercado).
• Summary Display: Each summary is presented in a well-formatted markdown view that includes:
  • Key points and highlights from Brazilian news
  • Proper formatting for bullet points and sections
  • Timestamps for each historical summary
• Historical Archive: Users can access and compare summaries from different time periods using a select dropdown, with summaries cached for quick access.
• Auto-Refresh: The page revalidates every hour to ensure fresh content.

# AI Provider Configuration (Choose one)
GROQ_API_KEY=<your-groq-api-key>        # If using Groq
OPENROUTER_API_KEY=<your-openrouter-api-key>  # If using OpenRouter

# Social Media Integration
INSTAGRAM_ACCESS_TOKEN=<your-instagram-long-lived-access-token>
TWITTER_CLIENT_ID=<your-twitter-app-client-id>
TWITTER_CLIENT_SECRET=<your-twitter-client-secret>
FACEBOOK_APP_ID=<your-facebook-app-id>
FACEBOOK_APP_SECRET=<your-facebook-app-secret>
FACEBOOK_PAGE_ID=<your-facebook-page-id>
FACEBOOK_PAGE_ACCESS_TOKEN=<your-facebook-page-access-token>

# Cloudflare Services
CLOUDFLARE_API_TOKEN=<your-cloudflare-api-token>  # For Browser Rendering API
CLOUDFLARE_ACCOUNT_ID=<your-cloudflare-account-id>  # For Browser Rendering API

# Discord Integration
DISCORD_TOKEN=<your-discord-bot-token>
DISCORD_GUILD_ID=<your-guild-id>

# Authentication & Payments
NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=<your-clerk-publishable-key>
CLERK_SECRET_KEY=<your-clerk-secret-key>
STRIPE_SECRET_KEY=<your-stripe-secret-key>
STRIPE_PRICE_ID=<your-stripe-price-id>
STRIPE_WEBHOOK_SECRET=<your-stripe-webhook-secret>

# API Configuration
SERVER_API_URL=https://news.fasttakeoff.org  # Production URL
# Use http://localhost:8787 for local development

# Optional Services
GOOGLE_GEOCODING_API_KEY=<your-google-geocoding-api-key>  # For News Globe

# LinkedIn Integration (placeholder for future implementation)
# LINKEDIN_CLIENT_ID=<your-linkedin-client-id>
# LINKEDIN_CLIENT_SECRET=<your-linkedin-client-secret>

1. Report Generation Testing:

# If dev server is already running:
scripts/generate-reports.sh  # Generate test reports
scripts/check-truncation.sh  # Verify report truncation

# For full test including build:
scripts/full-test.sh

2. Local Development with Cloudflare:

# Start local development server
bun run dev

# Test with Cloudflare Worker
bun run preview:patch:test  # Required after major code changes

# Deploy to production
bun run deploy

The system aggregates and summarizes news from major Brazilian RSS feeds through a two-stage AI process:

1. Update Cycle:

   • Automatic hourly updates
   • Historical archives maintained in FEEDS_CACHE
   • Accessible via /brazil with timestamp and topic selection

2. AI Processing:

   • Stage 1: Curation of relevant news based on impact and verifiability
   • Stage 2: Structured summarization by topics (Giro Geral, Mercado)

1. RSS Feed Failures:

   • System continues with available feeds if some sources fail
   • Check logs for specific feed errors
   • Verify source URLs in configuration

2. Cache Issues:

   • Clear KV cache manually through Cloudflare dashboard
   • Local development uses separate cache
   • Cache automatically refreshes hourly

3. Report Generation Failures:

   • Check AI provider status and API keys
   • Verify Discord bot permissions
   • Review logs for specific error messages

1. Reports and Summaries:

   • GET /api/reports - Fetch cached reports
   • GET /api/summaries/list - List available news summaries
   • GET /api/summaries/[key] - Fetch specific summary by key
   • GET /api/translate - Translate report content

2. Cache Control:

   • All endpoints use stale-while-revalidate strategy
   • Default cache TTL: 1 hour
   • Force refresh with ?fresh=true parameter

• List View (/executive-orders): Displays a paginated and searchable list of executive orders. Initial data is fetched server-side, with client-side components handling search, filtering, and pagination of the loaded set. Each order is presented as a card linking to its detailed view.
• Detailed View (/executive-orders/[id]): Shows comprehensive information for a single executive order. This includes:
  • Metadata (title, date, EO number, citation, category).
  • Links to official documents (Federal Register HTML, PDF, full text).
  • AI-Generated Summary: An AI-powered summary of the order is generated on-demand (via /api/summarize) and displayed. Summaries are cached in the user's browser (localStorage) to improve subsequent load times.
  • Full text of the order (rendered from HTML).
  • Links to related executive orders, identified by parsing disposition notes.

The "Current Events" section provides insights into real-time information aggregated from monitored Discord channels.

• Main View (/current-events): Displays a dashboard of the latest generated reports, with one card per active Discord channel. This view allows users to search for specific topics across all reports and sort channels by activity, recency of reports, or channel name. Each card links to a channel-specific view.
• Channel Detail View (/current-events/[channelId]): Shows a chronological timeline of all reports generated for a specific Discord channel, grouped by date. Each report in the timeline links to its detailed view.
• Report Detail View (/current-events/[channelId]/[reportId]): This is the most granular view, presenting:
  • The full content of a specific generated report (headline, city, body). Server-rendered with initial data for SEO.
  • AI-Powered Translation: Users can translate the report content into multiple languages (English, Spanish, French, German, Portuguese) using an AI translation service (via /api/translate).
  • Source Messages: An accordion section displays the original Discord messages that were used as sources for generating the report. Each message (MessageItem.tsx) can show its content, embeds (titles, descriptions, fields, author), and media attachments (images/videos displayed using MediaPreview.tsx).

The application uses Clerk for user authentication. Users can sign up, sign in, and manage their profile. The profile page also integrates with Stripe to allow users to subscribe to a premium plan.

The AI provider for report generation is configurable via src/lib/config.ts (currently using Gemini 2.5 Flash via OpenRouter as the active provider, with Groq/Llama 4 Maverick as an alternative).

A new interactive feature that displays news reports as markers on a 3D globe. It fetches report data, geocodes the associated city, and visualizes it.

• Technology: Uses @react-three/fiber and @react-three/drei for 3D rendering.
• Data Sources: News reports from /api/reports, geocoding via /api/geocode.

The application can automatically post generated reports to social media platforms:

• Twitter: Uses TwitterService (src/lib/twitter-service.ts) to post reports as threaded tweets. It handles OAuth 2.0 for authentication, storing tokens in a Cloudflare KV namespace (AUTH_TOKENS).
• Instagram: Uses InstagramService (src/lib/instagram-service.ts) to post reports to an Instagram Business account. This includes dynamically generated images with the report headline overlaid and a caption with the full report details.
• Facebook: Uses FacebookService (src/lib/facebook-service.ts) to post reports to a Facebook Page. Posts include the full report content with hashtags and links.

The Instagram service uses a modern, integrated approach for image generation:

1. HTML Generation: Creates styled HTML directly in the service with the report headline
2. Screenshot Generation: Uses Cloudflare's Browser Rendering API to capture a 1080x1080 JPEG screenshot
3. R2 Storage: Stores the generated image in Cloudflare R2 bucket with 7-day retention
4. Instagram Posting: Uses the R2 public URL to post the image with caption to Instagram

Automation Note: Social media posting is automatically triggered as part of the report generation process, which runs on a schedule (see Scheduled Tasks below) and can also be manually triggered.

The application utilizes Cloudflare Workers' scheduled events (cron jobs) for background tasks, managed in src/lib/cron.ts. The wrangler.toml file defines the specific cron patterns:

• Message Updates: Fetches new messages from Discord channels and writes to D1. Triggered by the cron pattern "0,30 * * * *" (top of hour and half past). Uses D1 batch inserts (BATCH_SIZE=100) for ~100x speedup (~1-2 seconds vs ~110 seconds).
• Window Evaluation: Evaluates channels and generates reports based on activity patterns. Triggered by the cron pattern "15,45 * * * *" (quarter past and quarter to). Uses bulk D1 queries (2 queries vs ~40-50) to minimize subrequests (~80% reduction).
• Other Tasks: Executive order document fetching (EO_DOCUMENTS), MktNews summaries (MKTNEWS_SUMMARY), financial data updates (FINANCIAL_DATA_QUEUE) on various schedules.
• Manual Triggers: The cron handler also supports specific string identifiers (e.g., MESSAGES, WINDOW_EVALUATION) for on-demand triggering of these tasks via /api/trigger-cron.

Why Split Schedules: Each Cloudflare Worker invocation has a 1,000 subrequest quota. By splitting MESSAGES and WINDOW_EVALUATION into separate cron triggers, each task gets its own quota, avoiding 'Too many API requests by single worker invocation' errors.

The application leverages Cloudflare KV for selective caching (e.g., reports, channels) while Discord message history now lives entirely in D1. A CacheManager utility (src/lib/cache-utils.ts) provides a consistent interface for interacting with KV namespaces, supporting TTL-based expiration and background refresh (stale-while-revalidate) strategies.

Key KV namespaces used (defined in wrangler.toml):

• EXECUTIVE_ORDERS_CACHE: Caches individual executive orders and ID lookups.
• REPORTS_CACHE: Caches generated news reports.
• CHANNELS_CACHE: Caches filtered lists of Discord channels.
• SUBSCRIPTIONS_CACHE: Caches Stripe subscription data.
• AUTH_TOKENS: Stores OAuth tokens for Twitter API access.
• GEOCODE_CACHE: Caches geocoding results for News Globe.
• FEEDS_CACHE: Caches RSS feed data for Brazil News.

Key R2 buckets used (defined in wrangler.toml):

• INSTAGRAM_IMAGES: Stores generated Instagram post images with automatic 7-day cleanup.