Skip to content

Jejiang/Synapse-Migration-App

Repository files navigation

Maintainer

This project is linked to the GitHub profile: TheCloudClassroom

Synapse to Fabric Migration Portal

A full-stack web application to migrate Azure Synapse Analytics data engineering items to Microsoft Fabric.

Architecture

This application uses a full-stack architecture with:

  • Frontend: React + Vite + Fluent UI (runs on port 3000)
  • Backend: Express.js + MSAL Node (runs on port 3001)
  • Authentication: OAuth 2.0 Authorization Code flow with Azure AD/Entra ID

The backend handles all authentication and Azure API calls, making it more secure and scalable than a pure SPA approach.

Features

  • Secure Authentication: Server-side Azure AD authentication with session management
  • Migration Wizard: Step-by-step wizard to configure migration
  • Item Migration: Supports migrating:
    • Lake Databases → Fabric Lakehouses
    • Notebooks → Fabric Notebooks
    • Spark Job Definitions → Fabric Spark Job Definitions
    • Spark Pools → Fabric Environments (configuration)
  • Migration Report: Real-time progress tracking and detailed reports
  • Error Handling: Comprehensive error display and troubleshooting

Prerequisites

  • Node.js 18+
  • npm or yarn
  • Azure AD App Registration (see setup below)

Azure AD Setup

  1. Go to Azure Portal → Microsoft Entra ID → App registrations

  2. Create a new registration:

    • Name: Synapse-to-Fabric-Migration-App
    • Supported account types: Single tenant (or Multi-tenant for cross-organization)
    • Redirect URI: Add as Web platform: http://localhost:3001/auth/callback
  3. Create a Client Secret:

    • Go to Certificates & secrets → New client secret
    • Copy the secret value (you'll need this for the .env file)
  4. Add API Permissions:

    • Microsoft Graph: User.Read
    • Azure Service Management: user_impersonation
    • Power BI Service: Workspace.ReadWrite.All, Item.ReadWrite.All
    • Azure Synapse Analytics: user_impersonation
  5. Grant admin consent for the permissions (if required)

Installation

# Clone and navigate to the project
cd synapse-migration-app

# Install frontend dependencies
npm install

# Install backend dependencies
cd server
npm install
cd ..

Configuration

Backend Configuration (server/.env)

cd server
cp .env.example .env

Edit server/.env with your Azure AD app details:

CLIENT_ID=your-app-client-id
CLIENT_SECRET=your-client-secret
TENANT_ID=your-tenant-id
SESSION_SECRET=random-secure-string-for-sessions

Frontend Configuration (optional)

Create a .env file in the root if you need to change the API URL:

VITE_API_URL=http://localhost:3001

Running the Application

You need to run both the backend and frontend:

Terminal 1 - Backend Server

cd server
npm run dev

Server runs on: http://localhost:3001

Terminal 2 - Frontend

npm run dev

Frontend runs on: http://localhost:3000

Usage

  1. Open http://localhost:3000 in your browser
  2. Click "Sign in with Microsoft"
  3. You'll be redirected to Microsoft login
  4. After login, you'll return to the migration wizard
  5. Follow the 4-step wizard:
    • Overview: Read about the migration process
    • Source: Select your Azure subscription and Synapse workspace
    • Destination: Select your Fabric workspace
    • Finish: Review and start the migration

API Endpoints

Authentication

  • GET /auth/login - Initiate login flow
  • GET /auth/callback - OAuth callback
  • GET /auth/logout - Logout
  • GET /auth/status - Check auth status
  • GET /auth/me - Get current user

Azure Resources

  • GET /api/subscriptions - List Azure subscriptions
  • GET /api/subscriptions/:id/synapse-workspaces - List Synapse workspaces

Fabric

  • GET /api/fabric/workspaces - List Fabric workspaces
  • GET /api/fabric/workspaces/:id/capacity - Check workspace capacity
  • POST /api/fabric/workspaces/:id/lakehouses - Create lakehouse

Synapse

  • GET /api/synapse/:workspace/notebooks - List notebooks
  • GET /api/synapse/:workspace/spark-pools - List Spark pools
  • GET /api/synapse/:workspace/spark-job-definitions - List Spark job definitions
  • GET /api/synapse/:workspace/migration-counts - Get item counts

Migration

  • POST /api/migration/start - Start migration
  • GET /api/migration/:id/status - Get migration status
  • GET /api/migration/history - Get migration history
  • POST /api/migration/:id/cancel - Cancel migration

Production Deployment

Build

# Build frontend
npm run build

# Build backend
cd server
npm run build

Environment Variables (Production)

NODE_ENV=production
PORT=3001
CLIENT_ID=your-app-client-id
CLIENT_SECRET=your-client-secret
TENANT_ID=your-tenant-id
SESSION_SECRET=your-production-session-secret
APP_URL=https://your-production-domain.com
REDIRECT_URI=https://your-production-domain.com/auth/callback
POST_LOGOUT_REDIRECT_URI=https://your-production-domain.com

Update your Azure AD app registration with production redirect URIs.

Tech Stack

  • Frontend: React 18, TypeScript, Vite, Fluent UI v9, React Router v6
  • Backend: Express.js, MSAL Node, express-session
  • Authentication: Azure AD OAuth 2.0 (Confidential Client)
  • APIs: Azure Resource Manager, Microsoft Fabric REST API, Azure Synapse REST API

Edit .env with your Azure AD app details:

VITE_CLIENT_ID=your-app-client-id
VITE_TENANT_ID=your-tenant-id
VITE_AUTHORITY=https://login.microsoftonline.com/your-tenant-id
VITE_REDIRECT_URI=http://localhost:3000

Project Structure

src/
├── components/
│   ├── layout/         # Header, Sidebar, Layout
│   ├── wizard/         # Migration wizard steps
│   └── report/         # Migration report components
├── config/
│   └── authConfig.ts   # MSAL configuration
├── context/
│   └── WizardContext.tsx # Wizard state management
├── pages/
│   ├── HomePage.tsx    # Landing page
│   └── ReportPage.tsx  # Migration report page
├── services/
│   ├── azureService.ts   # Azure Resource Manager API
│   ├── fabricService.ts  # Fabric REST API
│   └── synapseService.ts # Synapse REST API
├── styles/
│   ├── wizardStyles.ts   # Wizard component styles
│   └── reportStyles.ts   # Report component styles
├── types/
│   └── migration.ts    # TypeScript types
├── App.tsx             # Main app component
└── main.tsx            # Entry point

Available Scripts

  • npm run dev - Start development server
  • npm run build - Build for production
  • npm run preview - Preview production build
  • npm run lint - Run ESLint

API Endpoints Used

Azure Resource Manager

  • GET /subscriptions - List subscriptions
  • GET /subscriptions/{id}/providers/Microsoft.Synapse/workspaces - List Synapse workspaces

Fabric REST API

  • GET /v1/workspaces - List Fabric workspaces
  • POST /v1/workspaces/{id}/lakehouses - Create Lakehouse
  • POST /v1/workspaces/{id}/sparkJobDefinitions - Create Spark Job Definition

Synapse REST API

  • GET /bigDataPools - List Spark pools
  • GET /notebooks - List notebooks
  • GET /sparkJobDefinitions - List Spark job definitions

License

MIT

About

Synapse to Fabric Migration (Spark)

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published